Methods and Systems for Presenting Video Feeds

ABSTRACT

The disclosed embodiments include a computing system with a display, processor(s), and memory. The computing system displays a video monitoring user interface. The user interface includes a first region for displaying live and/or recorded video from a video camera of a smart home environment, and a second region for displaying an event timeline. The event timeline includes event indicators corresponding to respective events detected within the smart home environment. The video monitoring user interface includes a live video affordance for displaying live video from the video camera. In response to a user selection of an event indicator, the computing system: (1) requests a video feed of the corresponding detected event; and (2) displays the requested video feed in the first region. In response to a user selection of the live video affordance, the computing system: (1) requests the live video; and (2) displays the live video in the first region.

PRIORITY CLAIM AND RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/897,077, filed Feb. 14, 2018, entitled “Method and System forDisplaying Recorded and Live Video Feeds,” which is a continuation ofU.S. patent application Ser. No. 15/203,546, filed Jul. 6, 2016,entitled “Video Monitoring User Interface for Displaying Motion EventsFeed,” now U.S. Pat. No. 9,940,523, issued Apr. 10, 2018, which is acontinuation of U.S. patent application Ser. No. 15/173,419, filed Jun.3, 2016, entitled “Method and System for Categorizing Detected MotionEvents,” now U.S. Pat. No. 9,479,822, issued Oct. 25, 2016, which is acontinuation of U.S. patent application Ser. No. 14/510,042, filed Oct.8, 2014, entitled “Method and System for Categorizing Detected MotionEvents,” now U.S. Pat. No. 9,420,331, issued Aug. 16, 2016, which claimspriority to U.S. Provisional Patent Application No. 62/021,620, filedJul. 7, 2014, entitled “Activity Recognition and Video Filtering,” andU.S. Provisional Patent Application No. 62/057,991, filed Sep. 30, 2014,entitled “Method and System for Video Monitoring.” The content of eachof these applications is hereby incorporated by reference in itsentirety.

This application is related to U.S. Design patent application Ser. No.29/504,605, filed Oct. 7, 2014, entitled “Video Monitoring UserInterface with Event Timeline and Display of Multiple Preview Windows AtUser-Selected Event Marks,” now U.S. Pat. No. D782,495, issued Mar. 28,2017, U.S. patent application Ser. No. 15/202,494, filed Jul. 5, 2016,entitled “Method and System for Displaying Recorded and Live VideoFeeds,” now U.S. Pat. No. 9,602,860, issued Mar. 21, 2017, U.S. patentapplication Ser. No. 15/202,503, filed Jul. 5, 2016, entitled “Methodand System for Detecting and Presenting a New Event in a Video Feed,”now U.S. Pat. No. 9,609,380, issued Mar. 28, 2017, and U.S. patentapplication Ser. No. 15/203,557, filed Jul. 6, 2016, entitled “Methodand System for Detecting and Presenting Video Feed,” now U.S. Pat. No.9,674,570, issued Jun. 6, 2017, each of which is hereby incorporated byreference in its entirety.

TECHNICAL FIELD

The disclosed implementations relate generally to video monitoring,including, but not limited, to monitoring and reviewing motion events ina video stream.

BACKGROUND

Video surveillance produces a large amount of continuous video data overthe course of hours, days, and even months. Such video data includesmany long and uneventful portions that are of no significance orinterest to a reviewer. In some existing video surveillance systems,motion detection is used to trigger alerts or video recording. However,using motion detection as the only means for selecting video segmentsfor user review may still produce too many video segments that are of nointerest to the reviewer. For example, some detected motions aregenerated by normal activities that routinely occur at the monitoredlocation, and it is tedious and time consuming to manually scan throughall of the normal activities recorded on video to identify a smallnumber of activities that warrant special attention. In addition, whenthe sensitivity of the motion detection is set too high for the locationbeing monitored, trivial movements (e.g., movements of tree leaves,shifting of the sunlight, etc.) can account for a large amount of videobeing recorded and/or reviewed. On the other hand, when the sensitivityof the motion detection is set too low for the location being monitored,the surveillance system may fail to record and present video data onsome important and useful events.

It is a challenge to identify meaningful segments of the video streamand to present them to the reviewer in an efficient, intuitive, andconvenient manner. Human-friendly techniques for discovering andpresenting motion events of interest both in real-time or at a latertime are in great need.

SUMMARY

Accordingly, there is a need for video processing with more efficientand intuitive motion event identification, categorization, andpresentation. Such methods optionally complement or replace conventionalmethods for monitoring and reviewing motion events in a video stream.

In some implementations, a method of displaying indicators for motionevents on an event timeline is performed at an electronic device (e.g.,an electronic device 166, FIG. 1; or a client device 504, FIGS. 5 and 7)with one or more processors, memory, and a display. The method includesdisplaying a video monitoring user interface on the display including acamera feed from a camera located remotely from the client device in afirst region of the video monitoring user interface and an eventtimeline in a second region of the video monitoring user interface,where the event timeline includes a plurality of event indicators for aplurality of motion events previously detected by the camera. The methodincludes associating a newly created first category with a set ofsimilar motion events from among the plurality of motion eventspreviously detected by the camera. In response to associating the firstcategory with the first set of similar motion events, the methodincludes changing at least one display characteristic for a first set ofpre-existing event indicators from among the plurality of eventindicators on the event timeline that correspond to the first category,where the first set of pre-existing event indicators correspond to theset of similar motion events.

In some implementations, a method of editing event categories isperformed at an electronic device (e.g., the electronic device 166, FIG.1; or the client device 504, FIGS. 5 and 7) with one or more processors,memory, and a display. The method includes displaying a video monitoringuser interface on the display with a plurality of user interfaceelements associated one or more recognized activities. The methodincludes detecting a user input selecting a respective user interfaceelement from the plurality of user interface elements in the videomonitoring user interface, the respective user interface element beingassociated with a respective event category of the one or morerecognized event categories. In response to detecting the user input,the method includes displaying an editing user interface for therespective event category on the display with a plurality of animatedrepresentations in a first region of the editing user interface, wherethe plurality of animated representations correspond to a plurality ofpreviously captured motion events assigned to the respective eventcategory.

In some implementations, a method of categorizing a detected motionevent is performed at a computing system (e.g., the client device 504,FIGS. 5 and 7; the video server system 508, FIGS. 5-6; or a combinationthereof) with one or more processors and memory. The method includesdisplaying a video monitoring user interface on the display including avideo feed from a camera located remotely from the client device in afirst region of the video monitoring user interface and an eventtimeline in a second region of the video monitoring user interface,where the event timeline includes one or more event indicatorscorresponding to one or more motion events previously detected by thecamera. The method includes detecting a motion event and determining oneor more characteristics for the motion event. In accordance with adetermination that the one or more determined characteristics for themotion event satisfy one or more criteria for a respective eventcategory, the method includes: assigning the motion event to therespective category; and displaying an indicator for the detected motionevent on the event timeline with a display characteristic correspondingto the respective category.

In some implementations, a method of generating a smart time-lapse videoclip is performed at an electronic device (e.g., the electronic device166, FIG. 1; or the client device 504, FIGS. 5 and 7) with one or moreprocessors, memory, and a display. The method includes displaying avideo monitoring user interface on the display including a video feedfrom a camera located remotely from the client device in a first regionof the video monitoring user interface and an event timeline in a secondregion of the video monitoring user interface, where the event timelineincludes a plurality of event indicators for a plurality of motionevents previously detected by the camera. The method includes detectinga first user input selecting a portion of the event timeline, where theselected portion of the event timeline includes a subset of theplurality of event indicators on the event timeline. In response to thefirst user input, the method includes causing generation of a time-lapsevideo clip of the selected portion of the event timeline. The methodincludes displaying the time-lapse video clip of the selected portion ofthe event timeline, where motion events corresponding to the subset ofthe plurality of event indicators are played at a slower speed than theremainder of the selected portion of the event timeline.

In some implementations, a method of performing client-side zooming of aremote video feed is performed at an electronic device (e.g., theelectronic device 166, FIG. 1; or the client device 504, FIGS. 5 and 7)with one or more processors, memory, and a display. The method includesreceiving a first video feed from a camera located remotely from theclient device with a first field of view and displaying, on the display,the first video feed in a video monitoring user interface. The methodincludes detecting a first user input to zoom in on a respective portionof the first video feed and, in response to detecting the first userinput, performing a software zoom function on the respective portion ofthe first video feed to display the respective portion of the firstvideo feed in a first resolution. The method includes determining acurrent zoom magnification of the software zoom function and coordinatesof the respective portion of the first video feed and sending a commandto the camera to perform a hardware zoom function on the respectiveportion according to the current zoom magnification and the coordinatesof the respective portion of the first video feed. The method includesreceiving a second video feed from the camera with a second field ofview different from the first field of view, where the second field ofview corresponds to the respective portion and displaying, on thedisplay, the second video feed in the video monitoring user interface,where the second video feed is displayed in a second resolution that ishigher than the first resolution.

In accordance with some implementations, a method of processing a videostream is performed at a computing system having one or more processorsand memory (e.g., the camera 118, FIGS. 5 and 8; the video system server508, FIGS. 5-6; a combination thereof). The method includes processingthe video stream to detect a start of a first motion event candidate inthe video stream, In response to detecting the start of the first motionevent candidate in the video stream, the method includes initiatingevent recognition processing on a first video segment associated withthe start of the first motion event candidate, where initiating theevent recognition processing further includes: determining a motiontrack of a first object identified in the first video segment;generating a representative motion vector for the first motion eventcandidate based on the respective motion track of the first object; andsending the representative motion vector for the first motion eventcandidate to an event categorizer, where the event categorizer assigns arespective motion event category to the first motion event candidatebased on the representative motion vector of the first motion eventcandidate.

In accordance with some implementations, a method of categorizing amotion event candidate is performed at a server (e.g., the video serversystem 508, FIGS. 5-6) having one or more processors and memory. Themethod includes obtaining a respective motion vector for each of aseries of motion event candidates in real-time as said each motion eventcandidate is detected in a live video stream. In response to receivingthe respective motion vector for each of the series of motion eventcandidates, the method includes determining a spatial relationshipbetween the respective motion vector of said each motion event candidateto one or more existing clusters established based on a plurality ofpreviously processed motion vectors. In accordance with a determinationthat the respective motion vector of a first motion event candidate ofthe series of motion event candidates falls within a respective range ofat least a first existing cluster of the one or more existing clusters,the method includes assigning the first motion event candidate to atleast a first event category associated with the first existing cluster.

In accordance with some implementations, a method of facilitating reviewof a video recording is performed at a server (e.g., the video serversystem 508, FIGS. 5-6) having one or more processors and memory. Themethod includes identifying a plurality of motion events from a videorecording, wherein each of the motion events corresponds to a respectivevideo segment along a timeline of the video recording and identifies atleast one object in motion within a scene depicted in the videorecording. The method includes: storing a respective event mask for eachof the plurality of motion events identified in the video recording, therespective event mask including an aggregate of motion pixels associatedwith the at least one object in motion over multiple frames of themotion event; and receiving a definition of a zone of interest withinthe scene depicted in the video recording. In response to receiving thedefinition of the zone of interest, the method includes: determining,for each of the plurality of motion events, whether the respective eventmask of the motion event overlaps with the zone of interest by at leasta predetermined overlap factor; and identifying one or more events ofinterest from the plurality of motion events, where the respective eventmask of each of the identified events of interest is determined tooverlap with the zone of interest by at least the predetermined overlapfactor.

In accordance with some implementations, a method of monitoring selectedzones in a scene depicted in a video stream is performed at a server(e.g., the video server system 508, FIGS. 5-6) having one or moreprocessors and memory. The method includes receiving a definition of azone of interest within the scene depicted in the video steam. Inresponse to receiving the definition of the zone of interest, the methodincludes: determining, for each motion event detected in the videostream, whether a respective event mask of the motion event overlapswith the zone of interest by at least a predetermined overlap factor;and identifying the motion event as an event of interest associated withthe zone of interest in accordance with a determination that therespective event mask of the motion event overlaps with the zone ofinterest by at least the predetermined overlap factor.

In some implementations, a computing system (e.g., the video serversystem 508, FIGS. 5-6; the client device 504, FIGS. 5 and 7; or acombination thereof) includes one or more processors and memory storingone or more programs for execution by the one or more processors, andthe one or more programs include instructions for performing, orcontrolling performance of, the operations of any of the methodsdescribed herein. In some implementations, a non-transitory computerreadable storage medium stores one or more programs, where the one ormore programs include instructions, which, when executed by a computingsystem (e.g., the video server system 508, FIGS. 5-6; the client device504, FIGS. 5 and 7; or a combination thereof) with one or moreprocessors, cause the computing device to perform, or controlperformance of, the operations of any of the methods described herein.In some implementations, a computing system (e.g., the video serversystem 508, FIGS. 5-6; the client device 504, FIGS. 5 and 7; or acombination thereof) includes means for performing, or controllingperformance of, the operations of any of the methods described herein.

Thus, computing systems are provided with more efficient methods formonitoring and facilitating review of motion events in a video stream,thereby increasing the effectiveness, efficiency, and user satisfactionwith such systems. Such methods may complement or replace conventionalmethods for motion event monitoring and presentation.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described implementations,reference should be made to the Description of Implementations below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1 is a representative smart home environment in accordance withsome implementations.

FIG. 2 is a block diagram illustrating a representative networkarchitecture that includes a smart home network in accordance with someimplementations.

FIG. 3 illustrates a network-level view of an extensible devices andservices platform with which the smart home environment of FIG. 1 isintegrated, in accordance with some implementations.

FIG. 4 illustrates an abstracted functional view of the extensibledevices and services platform of FIG. 3, with reference to a processingengine as well as devices of the smart home environment, in accordancewith some implementations.

FIG. 5 is a representative operating environment in which a video serversystem interacts with client devices and video sources in accordancewith some implementations.

FIG. 6 is a block diagram illustrating a representative video serversystem in accordance with some implementations.

FIG. 7 is a block diagram illustrating a representative client device inaccordance with some implementations.

FIG. 8 is a block diagram illustrating a representative video capturingdevice (e.g., a camera) in accordance with some implementations.

FIGS. 9A-9BB illustrate example user interfaces on a client device formonitoring and reviewing motion events in accordance with someimplementations.

FIG. 10 illustrates a flow diagram of a process for performingclient-side zooming of a remote video feed in accordance with someimplementations.

FIG. 11A illustrates example system architecture and processing pipelinefor video monitoring in accordance with some implementations.

FIG. 11B illustrates techniques for motion event detection and falsepositive removal in video monitoring in accordance with someimplementations.

FIG. 11C illustrates an example motion mask and an example event maskgenerated based on video data in accordance with some implementations.

FIG. 11D illustrates a process for learning event categories andcategorizing motion events in accordance with some implementations.

FIG. 11E illustrates a process for identifying an event of interestbased on selected zones of interest in accordance with someimplementations.

FIGS. 12A-12B illustrate a flowchart diagram of a method of displayingindicators for motion events on an event timeline in accordance withsome implementations.

FIGS. 13A-13B illustrate a flowchart diagram of a method of editingevent categories in accordance with some implementations.

FIGS. 14A-14B illustrate a flowchart diagram of a method ofautomatically categorizing a detected motion event in accordance withsome implementations.

FIGS. 15A-15C illustrate a flowchart diagram of a method of generating asmart time-lapse video clip in accordance with some implementations.

FIGS. 16A-16B illustrate a flowchart diagram of a method of performingclient-side zooming of a remote video feed in accordance with someimplementations.

FIGS. 17A-17D illustrate a flowchart diagram of a method of processing avideo stream for video monitoring in accordance with someimplementations.

FIGS. 18A-18D illustrate a flowchart diagram of a method of performingactivity recognition for video monitoring in accordance with someimplementations.

FIGS. 19A-19C illustrate a flowchart diagram of a method of facilitatingreview of a video recording in accordance with some implementations.

FIGS. 20A-20B illustrate a flowchart diagram of a method of providingcontext-aware zone monitoring on a video server system in accordancewith some implementations.

Like reference numerals refer to corresponding parts throughout theseveral views of the drawings.

DESCRIPTION OF IMPLEMENTATIONS

This disclosure provides example user interfaces and data processingsystems and methods for video monitoring.

Video-based surveillance and security monitoring of a premises generatesa continuous video feed that may last hours, days, and even months.Although motion-based recording triggers can help trim down the amountof video data that is actually recorded, there are a number of drawbacksassociated with video recording triggers based on simple motiondetection in the live video feed. For example, when motion detection isused as a trigger for recording a video segment, the threshold of motiondetection must be set appropriately for the scene of the video;otherwise, the recorded video may include many video segments containingtrivial movements (e.g., lighting change, leaves moving in the wind,shifting of shadows due to changes in sunlight exposure, etc.) that areof no significance to a reviewer. On the other hand, if the motiondetection threshold is set too high, video data on important movementsthat are too small to trigger the recording may be irreversibly lost.Furthermore, at a location with many routine movements (e.g., carspassing through in front of a window) or constant movements (e.g., ascene with a running fountain, a river, etc.), recording triggers basedon motion detection are rendered ineffective, because motion detectioncan no longer accurately select out portions of the live video feed thatare of special significance. As a result, a human reviewer has to siftthrough a large amount of recorded video data to identify a small numberof motion events after rejecting a large number of routine movements,trivial movements, and movements that are of no interest for a presentpurpose.

Due to at least the challenges described above, it is desirable to havea method that maintains a continuous recording of a live video feed suchthat irreversible loss of video data is avoided and, at the same time,augments simple motion detection with false positive suppression andmotion event categorization. The false positive suppression techniqueshelp to downgrade motion events associated with trivial movements andconstant movements. The motion event categorization techniques help tocreate category-based filters for selecting only the types of motionevents that are of interest for a present purpose. As a result, thereviewing burden on the reviewer may be reduced. In addition, as thepresent purpose of the reviewer changes in the future, the reviewer cansimply choose to review other types of motion events by selecting theappropriate motion categories as event filters.

In addition, in some implementations, event categories can also be usedas filters for real-time notifications and alerts. For example, when anew motion event is detected in a live video feed, the new motion eventis immediately categorized, and if the event category of the newlydetected mention event is a category of interest selected by a reviewer,a real-time notification or alert can be sent to the reviewer regardingthe newly detected motion event. In addition, if the new event isdetected in the live video feed as the reviewer is viewing a timeline ofthe video feed, the event indicator and the notification of the newevent will have an appearance or display characteristic associated withthe event category.

Furthermore, as the types of motion events occurring at differentlocations and settings can vary greatly, and there are potentially aninfinite number of event categories for all motion events collected atthe video server system (e.g., the video server system 508). Therefore,it may be undesirable to have a set of fixed event categories from theoutset to categorize motion events detected in all video feeds from allcamera locations for all users. As disclosed herein, in someimplementations, the motion event categories for the video stream fromeach camera are gradually established through machine learning, and arethus tailored to the particular setting and use of the video camera.

In addition, in some implementations, as new event categories aregradually discovered based on clustering of past motion events, theevent indicators for the past events in a newly discovered eventcategory are refreshed to reflect the newly discovered event category.In some implementations, a clustering algorithm with automatic phase outof old, inactive, and/or sparse categories is used to categorize motionevents. As a camera changes location, event categories that are nolonger active are gradually retired without manual input to keep themotion event categorization model current. In some implementations, userinput editing the assignment of past motion events into respective eventcategories is also taken into account for future event categoryassignment and new category creation.

Furthermore, for example, within the scene of a video feed, multipleobjects may be moving simultaneously. In some implementations, themotion track associated with each moving object corresponds to arespective motion event candidate, such that the movement of thedifferent objects in the same scene may be assigned to different motionevent categories.

In general, motion events may occur in different regions of a scene atdifferent times. Out of all the motion events detected within a scene ofa video stream over time, a reviewer may only be interested in motionevents that occurred within or entered a particular zone of interest inthe scene. In addition, the zones of interest may not be known to thereviewer and/or the video server system until long after one or moremotion events of interest have occurred within the zones of interest.For example, a parent may not be interested in activities centeredaround a cookie jar until after some cookies have mysteriously gonemissing. Furthermore, the zones of interest in the scene of a video feedcan vary for a reviewer over time depending on a present purpose of thereviewer. For example, the parent may be interested in seeing allactivities that occurred around the cookie jar one day when some cookieshave gone missing, and the parent may be interested in seeing allactivities that occurred around a mailbox the next day when someexpected mail has gone missing. Accordingly, in some implementations,the techniques disclosed herein allow a reviewer to define and createone or more zones of interest within a static scene of a video feed, andthen use the created zones of interest to retroactively identify allpast motion events (or all motion events within a particular past timewindow) that have touched or entered the zones of interest. Theidentified motion events are optionally presented to the user in atimeline or in a list. In some implementations, real-time alerts for anynew motion events that touch or enter the zones of interest are sent tothe reviewer. The ability to quickly identify and retrieve past motionevents that are associated with a newly created zone of interestaddresses the drawbacks of conventional zone monitoring techniques wherethe zones of interest need to be defined first based on a certain degreeof guessing and anticipation that may later prove to be inadequate orwrong, and where only future events (as opposed to both past and futureevents) within the zones of interest can be identified.

Furthermore, when detecting new motion events that have touched orentered some zone(s) of interest, the event detection is based on themotion information collected from the entire scene, rather than justwithin the zone(s) of interest. In particular, aspects of motiondetection, motion object definition, motion track identification, falsepositive suppression, and event categorization are all based on imageinformation collected from the entire scene, rather than just withineach zone of interest. As a result, context around the zones of interestis taken into account when monitoring events within the zones ofinterest. Thus, the accuracy of event detection and categorization maybe improved as compared to conventional zone monitoring techniques thatperform all calculations with image data collected only within the zonesof interest.

Other aspects of event monitoring and review for video data aredisclosed, including system architecture, data processing pipeline,event categorization, user interfaces for editing and reviewing pastevents (e.g., event timeline, retroactive coloring of event indicators,event filters based on event categories and zones of interest, and smarttime-lapse video summary), notifying new events (e.g., real-time eventpop-ups), creating zones of interest, and controlling camera's operation(e.g., changing video feed focus and resolution), and the like.Advantages of these and other aspects will be discussed in more detaillater in the present disclosure or will be apparent to persons skilledin the art in light of the disclosure provided herein.

Below, FIGS. 1-4 provide an overview of exemplary smart home devicenetworks and capabilities. FIGS. 5-8 provide a description of thesystems and devices participating in the video monitoring. FIGS. 9A-9BBillustrate exemplary user interfaces for reviewing motion events (e.g.,user interfaces including event timelines, event notifications, andevent categories), editing event categories (e.g., user interface forediting motion events assigned to a particular category), and settingvideo monitoring preferences (e.g., user interfaces for creating andselecting zones of interest, setting zone monitoring triggers, selectingevent filters, changing camera operation state, etc.). FIG. 10illustrates the interaction between devices to alter a camera operationstate (e.g., zoom and data transmission). FIGS. 11A-11E illustrate dataprocessing techniques supporting the video monitoring and event reviewcapabilities described herein. FIGS. 12A-12B illustrate a flowchartdiagram of a method of displaying indicators for motion events on anevent timeline in accordance with some implementations. FIGS. 13A-13Billustrate a flowchart diagram of a method of editing event categoriesin accordance with some implementations. FIGS. 14A-14B illustrate aflowchart diagram of a method of automatically categorizing a detectedmotion event in accordance with some implementations. FIGS. 15A-15Cillustrate a flowchart diagram of a method of generating a smarttime-lapse video clip in accordance with some implementations. FIGS.16A-16B illustrate a flowchart diagram of a method of performingclient-side zooming of a remote video feed in accordance with someimplementations. FIGS. 17A-20B illustrate flowchart diagrams of methodsfor video monitoring and event review described herein. The userinterfaces in FIGS. 9A-9BB are used to illustrate the processes and/ormethods in FIGS. 10, 12A-12B, 13A-13B, 14A-14B, 15A-15C, and 16A-16B,and provide frontend examples and context for the backend processesand/or methods in FIGS. 11A-11E, 17A-17D, 18A-18D, 19A-19C, and 20A-20B.

Reference will now be made in detail to implementations, examples ofwhich are illustrated in the accompanying drawings. In the followingdetailed description, numerous specific details are set forth in orderto provide a thorough understanding of the various describedimplementations. However, it will be apparent to one of ordinary skillin the art that the various described implementations may be practicedwithout these specific details. In other instances, well-known methods,procedures, components, circuits, and networks have not been describedin detail so as not to unnecessarily obscure aspects of theimplementations.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first userinterface could be termed a second user interface, and, similarly, asecond user interface could be termed a first user interface, withoutdeparting from the scope of the various described implementations. Thefirst user interface and the second user interface are both userinterfaces, but they are not the same user interface.

The terminology used in the description of the various describedimplementations herein is for the purpose of describing particularimplementations only and is not intended to be limiting. As used in thedescription of the various described implementations and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting”or “in accordance with a determination that,” depending on the context.Similarly, the phrase “if it is determined” or “if [a stated conditionor event] is detected” is, optionally, construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event]” or “in accordance with a determination that [astated condition or event] is detected,” depending on the context.

It is to be appreciated that “smart home environments” may refer tosmart environments for homes such as a single-family house, but thescope of the present teachings is not so limited. The present teachingsare also applicable, without limitation, to duplexes, townhomes,multi-unit apartment buildings, hotels, retail stores, office buildings,industrial buildings, and more generally any living space or work space.

It is also to be appreciated that while the terms user, customer,installer, homeowner, occupant, guest, tenant, landlord, repair person,and the like may be used to refer to the person or persons acting in thecontext of some particularly situations described herein, thesereferences do not limit the scope of the present teachings with respectto the person or persons who are performing such actions. Thus, forexample, the terms user, customer, purchaser, installer, subscriber, andhomeowner may often refer to the same person in the case of asingle-family residential dwelling, because the head of the household isoften the person who makes the purchasing decision, buys the unit, andinstalls and configures the unit, and is also one of the users of theunit. However, in other scenarios, such as a landlord-tenantenvironment, the customer may be the landlord with respect to purchasingthe unit, the installer may be a local apartment supervisor, a firstuser may be the tenant, and a second user may again be the landlord withrespect to remote control functionality. Importantly, while the identityof the person performing the action may be germane to a particularadvantage provided by one or more of the implementations, such identityshould not be construed in the descriptions that follow as necessarilylimiting the scope of the present teachings to those particularindividuals having those particular identities.

FIG. 1 is a representative smart home environment in accordance withsome implementations. Smart home environment 100 includes a structure150, which is optionally a house, office building, garage, or mobilehome. It will be appreciated that devices may also be integrated into asmart home environment 100 that does not include an entire structure150, such as an apartment, condominium, or office space. Further, thesmart home environment may control and/or be coupled to devices outsideof the actual structure 150. Indeed, several devices in the smart homeenvironment need not be physically within the structure 150. Forexample, a device controlling a pool heater 114 or irrigation system 116may be located outside of structure 150.

The depicted structure 150 includes a plurality of rooms 152, separatedat least partly from each other via walls 154. The walls 154 may includeinterior walls or exterior walls. Each room may further include a floor156 and a ceiling 158. Devices may be mounted on, integrated with and/orsupported by a wall 154, floor 156 or ceiling 158.

In some implementations, the smart home environment 100 includes aplurality of devices, including intelligent, multi-sensing,network-connected devices, that integrate seamlessly with each other ina smart home network (e.g., 202 FIG. 2) and/or with a central server ora cloud-computing system to provide a variety of useful smart homefunctions. The smart home environment 100 may include one or moreintelligent, multi-sensing, network-connected thermostats 102(hereinafter referred to as “smart thermostats 102”), one or moreintelligent, network-connected, multi-sensing hazard detection units 104(hereinafter referred to as “smart hazard detectors 104”), and one ormore intelligent, multi-sensing, network-connected entryway interfacedevices 106 (hereinafter referred to as “smart doorbells 106”). In someimplementations, the smart thermostat 102 detects ambient climatecharacteristics (e.g., temperature and/or humidity) and controls a HVACsystem 103 accordingly. The smart hazard detector 104 may detect thepresence of a hazardous substance or a substance indicative of ahazardous substance (e.g., smoke, fire, and/or carbon monoxide). Thesmart doorbell 106 may detect a person's approach to or departure from alocation (e.g., an outer door), control doorbell functionality, announcea person's approach or departure via audio or visual means, and/orcontrol settings on a security system (e.g., to activate or deactivatethe security system when occupants go and come).

In some implementations, the smart home environment 100 includes one ormore intelligent, multi-sensing, network-connected wall switches 108(hereinafter referred to as “smart wall switches 108”), along with oneor more intelligent, multi-sensing, network-connected wall pluginterfaces 110 (hereinafter referred to as “smart wall plugs 110”). Thesmart wall switches 108 may detect ambient lighting conditions, detectroom-occupancy states, and control a power and/or dim state of one ormore lights. In some instances, smart wall switches 108 may also controla power state or speed of a fan, such as a ceiling fan. The smart wallplugs 110 may detect occupancy of a room or enclosure and control supplyof power to one or more wall plugs (e.g., such that power is notsupplied to the plug if nobody is at home).

In some implementations, the smart home environment 100 of FIG. 1includes a plurality of intelligent, multi-sensing, network-connectedappliances 112 (hereinafter referred to as “smart appliances 112”), suchas refrigerators, stoves, ovens, televisions, washers, dryers, lights,stereos, intercom systems, garage-door openers, floor fans, ceilingfans, wall air conditioners, pool heaters, irrigation systems, securitysystems, space heaters, window AC units, motorized duct vents, and soforth. In some implementations, when plugged in, an appliance mayannounce itself to the smart home network, such as by indicating whattype of appliance it is, and it may automatically integrate with thecontrols of the smart home. Such communication by the appliance to thesmart home may be facilitated by either a wired or wirelesscommunication protocol. The smart home may also include a variety ofnon-communicating legacy appliances 140, such as old conventionalwasher/dryers, refrigerators, and the like, which may be controlled bysmart wall plugs 110. The smart home environment 100 may further includea variety of partially communicating legacy appliances 142, such asinfrared (“IR”) controlled wall air conditioners or other IR-controlleddevices, which may be controlled by IR signals provided by the smarthazard detectors 104 or the smart wall switches 108.

In some implementations, the smart home environment 100 includes one ormore network-connected cameras 118 that are configured to provide videomonitoring and security in the smart home environment 100.

The smart home environment 100 may also include communication withdevices outside of the physical home but within a proximate geographicalrange of the home. For example, the smart home environment 100 mayinclude a pool heater monitor 114 that communicates a current pooltemperature to other devices within the smart home environment 100and/or receives commands for controlling the pool temperature.Similarly, the smart home environment 100 may include an irrigationmonitor 116 that communicates information regarding irrigation systemswithin the smart home environment 100 and/or receives controlinformation for controlling such irrigation systems.

By virtue of network connectivity, one or more of the smart home devicesof FIG. 1 may further allow a user to interact with the device even ifthe user is not proximate to the device. For example, a user maycommunicate with a device using a computer (e.g., a desktop computer,laptop computer, or tablet) or other portable electronic device (e.g., asmartphone) 166. A webpage or application may be configured to receivecommunications from the user and control the device based on thecommunications and/or to present information about the device'soperation to the user. For example, the user may view a current setpoint temperature for a device and adjust it using a computer. The usermay be in the structure during this remote communication or outside thestructure.

As discussed above, users may control the smart thermostat and othersmart devices in the smart home environment 100 using anetwork-connected computer or portable electronic device 166. In someexamples, some or all of the occupants (e.g., individuals who live inthe home) may register their device 166 with the smart home environment100. Such registration may be made at a central server to authenticatethe occupant and/or the device as being associated with the home and togive permission to the occupant to use the device to control the smartdevices in the home. An occupant may use their registered device 166 toremotely control the smart devices of the home, such as when theoccupant is at work or on vacation. The occupant may also use theirregistered device to control the smart devices when the occupant isactually located inside the home, such as when the occupant is sittingon a couch inside the home. It should be appreciated that instead of orin addition to registering the devices 166, the smart home environment100 may make inferences about which individuals live in the home and aretherefore occupants and which devices 166 are associated with thoseindividuals. As such, the smart home environment may “learn” who is anoccupant and permit the devices 166 associated with those individuals tocontrol the smart devices of the home.

In some implementations, in addition to containing processing andsensing capabilities, the devices 102, 104, 106, 108, 110, 112, 114,116, and/or 118 (collectively referred to as “the smart devices”) arecapable of data communications and information sharing with other smartdevices, a central server or cloud-computing system, and/or otherdevices that are network-connected. The required data communications maybe carried out using any of a variety of custom or standard wirelessprotocols (IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave,Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.) and/or any of avariety of custom or standard wired protocols (CAT6 Ethernet, HomePlug,etc.), or any other suitable communication protocol, includingcommunication protocols not yet developed as of the filing date of thisdocument.

In some implementations, the smart devices serve as wireless or wiredrepeaters. For example, a first one of the smart devices communicateswith a second one of the smart devices via a wireless router. The smartdevices may further communicate with each other via a connection to oneor more networks 162 such as the Internet. Through the one or morenetworks 162, the smart devices may communicate with a smart homeprovider server system 164 (also called a central server system and/or acloud-computing system herein). In some implementations, the smart homeprovider server system 164 may include multiple server systems eachdedicated to data processing associated with a respective subset of thesmart devices (e.g., a video server system may be dedicated to dataprocessing associated with camera(s) 118). The smart home providerserver system 164 may be associated with a manufacturer, support entity,or service provider associated with the smart device. In someimplementations, a user is able to contact customer support using asmart device itself rather than needing to use other communicationmeans, such as a telephone or Internet-connected computer. In someimplementations, software updates are automatically sent from the smarthome provider server system 164 to smart devices (e.g., when available,when purchased, or at routine intervals).

FIG. 2 is a block diagram illustrating a representative networkarchitecture 200 that includes a smart home network 202 in accordancewith some implementations. In some implementations, one or more smartdevices 204 in the smart home environment 100 (e.g., the devices 102,104, 106, 108, 110, 112, 114, 116, and/or 118) combine to create a meshnetwork in the smart home network 202. In some implementations, the oneor more smart devices 204 in the smart home network 202 operate as asmart home controller. In some implementations, a smart home controllerhas more computing power than other smart devices. In someimplementations, a smart home controller processes inputs (e.g., fromthe smart device(s) 204, the electronic device 166, and/or the smarthome provider server system 164) and sends commands (e.g., to the smartdevice(s) 204 in the smart home network 202) to control operation of thesmart home environment 100. In some implementations, some of the smartdevice(s) 204 in the mesh network are “spokesman” nodes (e.g., node204-1) and others are “low-powered” nodes (e.g., node 204-9). Some ofthe smart device(s) 204 in the smart home environment 100 are batterypowered, while others have a regular and reliable power source, such asby connecting to wiring (e.g., to 120V line voltage wires) behind thewalls 154 of the smart home environment. The smart devices that have aregular and reliable power source are referred to as “spokesman” nodes.These nodes are typically equipped with the capability of using awireless protocol to facilitate bidirectional communication with avariety of other devices in the smart home environment 100, as well aswith the central server or cloud-computing system 164. In someimplementations, one or more “spokesman” nodes operate as a smart homecontroller. On the other hand, the devices that are battery powered arereferred to as “low-power” nodes. These nodes tend to be smaller thanspokesman nodes and typically only communicate using wireless protocolsthat require very little power, such as Zigbee, 6LoWPAN, etc.

In some implementations, some low-power nodes are incapable ofbidirectional communication. These low-power nodes send messages, butthey are unable to “listen”. Thus, other devices in the smart homeenvironment 100, such as the spokesman nodes, cannot send information tothese low-power nodes.

As described, the spokesman nodes and some of the low-powered nodes arecapable of “listening.” Accordingly, users, other devices, and/or thecentral server or cloud-computing system 164 may communicate controlcommands to the low-powered nodes. For example, a user may use theportable electronic device 166 (e.g., a smartphone) to send commandsover the Internet to the central server or cloud-computing system 164,which then relays the commands to one or more spokesman nodes in thesmart home network 202. The spokesman nodes drop down to a low-powerprotocol to communicate the commands to the low-power nodes throughoutthe smart home network 202, as well as to other spokesman nodes that didnot receive the commands directly from the central server orcloud-computing system 164.

In some implementations, a smart nightlight 170 is a low-power node. Inaddition to housing a light source, the smart nightlight 170 houses anoccupancy sensor, such as an ultrasonic or passive IR sensor, and anambient light sensor, such as a photo resistor or a single-pixel sensorthat measures light in the room. In some implementations, the smartnightlight 170 is configured to activate the light source when itsambient light sensor detects that the room is dark and when itsoccupancy sensor detects that someone is in the room. In otherimplementations, the smart nightlight 170 is simply configured toactivate the light source when its ambient light sensor detects that theroom is dark. Further, in some implementations, the smart nightlight 170includes a low-power wireless communication chip (e.g., a ZigBee chip)that regularly sends out messages regarding the occupancy of the roomand the amount of light in the room, including instantaneous messagescoincident with the occupancy sensor detecting the presence of a personin the room. As mentioned above, these messages may be sent wirelessly,using the mesh network, from node to node (i.e., smart device to smartdevice) within the smart home network 202 as well as over the one ormore networks 162 to the central server or cloud-computing system 164.

Other examples of low-power nodes include battery-operated versions ofthe smart hazard detectors 104. These smart hazard detectors 104 areoften located in an area without access to constant and reliable powerand may include any number and type of sensors, such as smoke/fire/heatsensors, carbon monoxide/dioxide sensors, occupancy/motion sensors,ambient light sensors, temperature sensors, humidity sensors, and thelike. Furthermore, the smart hazard detectors 104 may send messages thatcorrespond to each of the respective sensors to the other devices and/orthe central server or cloud-computing system 164, such as by using themesh network as described above.

Examples of spokesman nodes include smart doorbells 106, smartthermostats 102, smart wall switches 108, and smart wall plugs 110.These devices 102, 106, 108, and 110 are often located near andconnected to a reliable power source, and therefore may include morepower-consuming components, such as one or more communication chipscapable of bidirectional communication in a variety of protocols.

In some implementations, the smart home environment 100 includes servicerobots 168 that are configured to carry out, in an autonomous manner,any of a variety of household tasks.

FIG. 3 illustrates a network-level view of an extensible devices andservices platform 300 with which the smart home environment 100 of FIG.1 is integrated, in accordance with some implementations. The extensibledevices and services platform 300 includes remote servers or cloudcomputing system 164. Each of the intelligent, network-connected devices102, 104, 106, 108, 110, 112, 114, 116, and 118 from FIG. 1 (identifiedsimply as “devices” in FIGS. 2-4) may communicate with the remoteservers or cloud computing system 164. For example, a connection to theone or more networks 162 may be established either directly (e.g., using3G/4G connectivity to a wireless carrier), or through a networkinterface 160 (e.g., a router, switch, gateway, hub, or an intelligent,dedicated whole-home control node), or through any combination thereof.

In some implementations, the devices and services platform 300communicates with and collects data from the smart devices of the smarthome environment 100. In addition, in some implementations, the devicesand services platform 300 communicates with and collects data from aplurality of smart home environments across the world. For example, thesmart home provider server system 164 collects home data 302 from thedevices of one or more smart home environments, where the devices mayroutinely transmit home data or may transmit home data in specificinstances (e.g., when a device queries the home data 302). Examplecollected home data 302 includes, without limitation, power consumptiondata, occupancy data, HVAC settings and usage data, carbon monoxidelevels data, carbon dioxide levels data, volatile organic compoundslevels data, sleeping schedule data, cooking schedule data, inside andoutside temperature humidity data, television viewership data, insideand outside noise level data, pressure data, video data, etc.

In some implementations, the smart home provider server system 164provides one or more services 304 to smart homes. Example services 304include, without limitation, software updates, customer support, sensordata collection/logging, remote access, remote or distributed control,and/or use suggestions (e.g., based on the collected home data 302) toimprove performance, reduce utility cost, increase safety, etc. In someimplementations, data associated with the services 304 is stored at thesmart home provider server system 164, and the smart home providerserver system 164 retrieves and transmits the data at appropriate times(e.g., at regular intervals, upon receiving a request from a user,etc.).

In some implementations, the extensible devices and the servicesplatform 300 includes a processing engine 306, which may be concentratedat a single server or distributed among several different computingentities without limitation. In some implementations, the processingengine 306 includes engines configured to receive data from the devicesof smart home environments (e.g., via the Internet and/or a networkinterface), to index the data, to analyze the data and/or to generatestatistics based on the analysis or as part of the analysis. In someimplementations, the analyzed data is stored as derived home data 308.

Results of the analysis or statistics may thereafter be transmitted backto the device that provided home data used to derive the results, toother devices, to a server providing a webpage to a user of the device,or to other non-smart device entities. In some implementations, usestatistics, use statistics relative to use of other devices, usepatterns, and/or statistics summarizing sensor readings are generated bythe processing engine 306 and transmitted. The results or statistics maybe provided via the one or more networks 162. In this manner, theprocessing engine 306 may be configured and programmed to derive avariety of useful information from the home data 302. A single servermay include one or more processing engines.

The derived home data 308 may be used at different granularities for avariety of useful purposes, ranging from explicit programmed control ofthe devices on a per-home, per-neighborhood, or per-region basis (forexample, demand-response programs for electrical utilities), to thegeneration of inferential abstractions that may assist on a per-homebasis (for example, an inference may be drawn that the homeowner hasleft for vacation and so security detection equipment may be put onheightened sensitivity), to the generation of statistics and associatedinferential abstractions that may be used for government or charitablepurposes. For example, processing engine 306 may generate statisticsabout device usage across a population of devices and send thestatistics to device users, service providers or other entities (e.g.,entities that have requested the statistics and/or entities that haveprovided monetary compensation for the statistics).

In some implementations, to encourage innovation and research and toincrease products and services available to users, the devices andservices platform 300 exposes a range of application programminginterfaces (APIs) 310 to third parties, such as charities 314,governmental entities 316 (e.g., the Food and Drug Administration or theEnvironmental Protection Agency), academic institutions 318 (e.g.,university researchers), businesses 320 (e.g., providing devicewarranties or service to related equipment, targeting advertisementsbased on home data), utility companies 324, and other third parties. TheAPIs 310 are coupled to and permit third-party systems to communicatewith the smart home provider server system 164, including the services304, the processing engine 306, the home data 302, and the derived homedata 308. In some implementations, the APIs 310 allow applicationsexecuted by the third parties to initiate specific data processing tasksthat are executed by the smart home provider server system 164, as wellas to receive dynamic updates to the home data 302 and the derived homedata 308.

For example, third parties may develop programs and/or applications,such as web applications or mobile applications, that integrate with thesmart home provider server system 164 to provide services andinformation to users. Such programs and applications may be, forexample, designed to help users reduce energy consumption, topreemptively service faulty equipment, to prepare for high servicedemands, to track past service performance, etc., and/or to performother beneficial functions or tasks.

FIG. 4 illustrates an abstracted functional view 400 of the extensibledevices and services platform 300 of FIG. 3, with reference to aprocessing engine 306 as well as devices of the smart home environment,in accordance with some implementations. Even though devices situated insmart home environments will have a wide variety of different individualcapabilities and limitations, the devices may be thought of as sharingcommon characteristics in that each device is a data consumer 402 (DC),a data source 404 (DS), a services consumer 406 (SC), and a servicessource 408 (SS). Advantageously, in addition to providing controlinformation used by the devices to achieve their local and immediateobjectives, the extensible devices and services platform 300 may also beconfigured to use the large amount of data that is generated by thesedevices. In addition to enhancing or optimizing the actual operation ofthe devices themselves with respect to their immediate functions, theextensible devices and services platform 300 may be directed to“repurpose” that data in a variety of automated, extensible, flexible,and/or scalable ways to achieve a variety of useful objectives. Theseobjectives may be predefined or adaptively identified based on, e.g.,usage patterns, device efficiency, and/or user input (e.g., requestingspecific functionality).

FIG. 4 shows the processing engine 306 as including a number ofprocessing paradigms 410. In some implementations, the processing engine306 includes a managed services paradigm 410 a that monitors and managesprimary or secondary device functions. The device functions may includeensuring proper operation of a device given user inputs, estimating that(e.g., and responding to an instance in which) an intruder is or isattempting to be in a dwelling, detecting a failure of equipment coupledto the device (e.g., a light bulb having burned out), implementing orotherwise responding to energy demand response events, and/or alerting auser of a current or predicted future event or characteristic. In someimplementations, the processing engine 306 includes anadvertising/communication paradigm 410 b that estimates characteristics(e.g., demographic information), desires and/or products of interest ofa user based on device usage. Services, promotions, products or upgradesmay then be offered or automatically provided to the user. In someimplementations, the processing engine 306 includes a social paradigm410 c that uses information from a social network, provides informationto a social network (for example, based on device usage), and/orprocesses data associated with user and/or device interactions with thesocial network platform. For example, a user's status as reported totheir trusted contacts on the social network may be updated to indicatewhen the user is home based on light detection, security systeminactivation or device usage detectors. As another example, a user maybe able to share device-usage statistics with other users. In yetanother example, a user may share HVAC settings that result in low powerbills and other users may download the HVAC settings to their smartthermostat 102 to reduce their power bills.

In some implementations, the processing engine 306 includes achallenges/rules/compliance/rewards paradigm 410 d that informs a userof challenges, competitions, rules, compliance regulations and/orrewards and/or that uses operation data to determine whether a challengehas been met, a rule or regulation has been complied with and/or areward has been earned. The challenges, rules, and/or regulations mayrelate to efforts to conserve energy, to live safely (e.g., reducingexposure to toxins or carcinogens), to conserve money and/or equipmentlife, to improve health, etc. For example, one challenge may involveparticipants turning down their thermostat by one degree for one week.Those participants that successfully complete the challenge arerewarded, such as with coupons, virtual currency, status, etc. Regardingcompliance, an example involves a rental-property owner making a rulethat no renters are permitted to access certain owner's rooms. Thedevices in the room having occupancy sensors may send updates to theowner when the room is accessed.

In some implementations, the processing engine 306 integrates orotherwise uses extrinsic information 412 from extrinsic sources toimprove the functioning of one or more processing paradigms. Theextrinsic information 412 may be used to interpret data received from adevice, to determine a characteristic of the environment near the device(e.g., outside a structure that the device is enclosed in), to determineservices or products available to the user, to identify a social networkor social-network information, to determine contact information ofentities (e.g., public-service entities such as an emergency-responseteam, the police or a hospital) near the device, to identify statisticalor environmental conditions, trends or other information associated witha home or neighborhood, and so forth.

FIG. 5 illustrates a representative operating environment 500 in which avideo server system 508 provides data processing for monitoring andfacilitating review of motion events in video streams captured by videocameras 118. As shown in FIG. 5, the video server system 508 receivesvideo data from video sources 522 (including cameras 118) located atvarious physical locations (e.g., inside homes, restaurants, stores,streets, parking lots, and/or the smart home environments 100 of FIG.1). Each video source 522 may be bound to one or more reviewer accounts,and the video server system 508 provides video monitoring data for thevideo source 522 to client devices 504 associated with the revieweraccounts. For example, the portable electronic device 166 is an exampleof the client device 504.

In some implementations, the smart home provider server system 164 or acomponent thereof serves as the video server system 508. In someimplementations, the video server system 508 is a dedicated videoprocessing server that provides video processing services to videosources and client devices 504 independent of other services provided bythe video server system 508.

In some implementations, each of the video sources 522 includes one ormore video cameras 118 that capture video and send the captured video tothe video server system 508 substantially in real-time. In someimplementations, each of the video sources 522 optionally includes acontroller device (not shown) that serves as an intermediary between theone or more cameras 118 and the video server system 508. The controllerdevice receives the video data from the one or more cameras 118,optionally, performs some preliminary processing on the video data, andsends the video data to the video server system 508 on behalf of the oneor more cameras 118 substantially in real-time. In some implementations,each camera has its own on-board processing capabilities to perform somepreliminary processing on the captured video data before sending theprocessed video data (along with metadata obtained through thepreliminary processing) to the controller device and/or the video serversystem 508.

As shown in FIG. 5, in accordance with some implementations, each of theclient devices 504 includes a client-side module 502. The client-sidemodule 502 communicates with a server-side module 506 executed on thevideo server system 508 through the one or more networks 162. Theclient-side module 502 provides client-side functionalities for theevent monitoring and review processing and communications with theserver-side module 506. The server-side module 506 provides server-sidefunctionalities for event monitoring and review processing for anynumber of client-side modules 502 each residing on a respective clientdevice 504. The server-side module 506 also provides server-sidefunctionalities for video processing and camera control for any numberof the video sources 522, including any number of control devices andthe cameras 118.

In some implementations, the server-side module 506 includes one or moreprocessors 512, a video storage database 514, an account database 516,an I/O interface to one or more client devices 518, and an I/O interfaceto one or more video sources 520. The I/O interface to one or moreclients 518 facilitates the client-facing input and output processingfor the server-side module 506. The account database 516 stores aplurality of profiles for reviewer accounts registered with the videoprocessing server, where a respective user profile includes accountcredentials for a respective reviewer account, and one or more videosources linked to the respective reviewer account. The I/O interface toone or more video sources 520 facilitates communications with one ormore video sources 522 (e.g., groups of one or more cameras 118 andassociated controller devices). The video storage database 514 storesraw video data received from the video sources 522, as well as varioustypes of metadata, such as motion events, event categories, eventcategory models, event filters, and event masks, for use in dataprocessing for event monitoring and review for each reviewer account.

Examples of a representative client device 504 include, but are notlimited to, a handheld computer, a wearable computing device, a personaldigital assistant (PDA), a tablet computer, a laptop computer, a desktopcomputer, a cellular telephone, a smart phone, an enhanced generalpacket radio service (EGPRS) mobile phone, a media player, a navigationdevice, a game console, a television, a remote control, a point-of-sale(POS) terminal, vehicle-mounted computer, an ebook reader, or acombination of any two or more of these data processing devices or otherdata processing devices.

Examples of the one or more networks 162 include local area networks(LAN) and wide area networks (WAN) such as the Internet. The one or morenetworks 162 are, optionally, implemented using any known networkprotocol, including various wired or wireless protocols, such asEthernet, Universal Serial Bus (USB), FIREWIRE, Long Term Evolution(LTE), Global System for Mobile Communications (GSM), Enhanced Data GSMEnvironment (EDGE), code division multiple access (CDMA), time divisionmultiple access (TDMA), Bluetooth, Wi-Fi, voice over Internet Protocol(VoIP), Wi-MAX, or any other suitable communication protocol.

In some implementations, the video server system 508 is implemented onone or more standalone data processing apparatuses or a distributednetwork of computers. In some implementations, the video server system508 also employs various virtual devices and/or services of third partyservice providers (e.g., third-party cloud service providers) to providethe underlying computing resources and/or infrastructure resources ofthe video server system 508. In some implementations, the video serversystem 508 includes, but is not limited to, a handheld computer, atablet computer, a laptop computer, a desktop computer, or a combinationof any two or more of these data processing devices or other dataprocessing devices.

The server-client environment 500 shown in FIG. 1 includes both aclient-side portion (e.g., the client-side module 502) and a server-sideportion (e.g., the server-side module 506). The division offunctionalities between the client and server portions of operatingenvironment 500 can vary in different implementations. Similarly, thedivision of functionalities between the video source 522 and the videoserver system 508 can vary in different implementations. For example, insome implementations, client-side module 502 is a thin-client thatprovides only user-facing input and output processing functions, anddelegates all other data processing functionalities to a backend server(e.g., the video server system 508). Similarly, in some implementations,a respective one of the video sources 522 is a simple video capturingdevice that continuously captures and streams video data to the videoserver system 508 without no or limited local preliminary processing onthe video data. Although many aspects of the present technology aredescribed from the perspective of the video server system 508, thecorresponding actions performed by the client device 504 and/or thevideo sources 522 would be apparent to ones skilled in the art withoutany creative efforts. Similarly, some aspects of the present technologymay be described from the perspective of the client device or the videosource, and the corresponding actions performed by the video serverwould be apparent to ones skilled in the art without any creativeefforts. Furthermore, some aspects of the present technology may beperformed by the video server system 508, the client device 504, and thevideo sources 522 cooperatively.

FIG. 6 is a block diagram illustrating the video server system 508 inaccordance with some implementations. The video server system 508,typically, includes one or more processing units (CPUs) 512, one or morenetwork interfaces 604 (e.g., including the I/O interface to one or moreclients 518 and the I/O interface to one or more video sources 520),memory 606, and one or more communication buses 608 for interconnectingthese components (sometimes called a chipset). The memory 606 includeshigh-speed random access memory, such as DRAM, SRAM, DDR RAM, or otherrandom access solid state memory devices; and, optionally, includesnon-volatile memory, such as one or more magnetic disk storage devices,one or more optical disk storage devices, one or more flash memorydevices, or one or more other non-volatile solid state storage devices.The memory 606, optionally, includes one or more storage devicesremotely located from the one or more processing units 512. The memory606, or alternatively the non-volatile memory within the memory 606,includes a non-transitory computer readable storage medium. In someimplementations, the memory 606, or the non-transitory computer readablestorage medium of the memory 606, stores the following programs,modules, and data structures, or a subset or superset thereof:

-   -   Operating system 610 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   Network communication module 612 for connecting the video server        system 508 to other computing devices (e.g., the client devices        504 and the video sources 522 including camera(s) 118) connected        to the one or more networks 162 via the one or more network        interfaces 604 (wired or wireless);    -   Server-side module 506, which provides server-side data        processing and functionalities for the event monitoring and        review, including but not limited to:        -   Account administration module 614 for creating reviewer            accounts, performing camera registration processing to            establish associations between video sources to their            respective reviewer accounts, and providing account            login-services to the client devices 504;        -   Video data receiving module 616 for receiving raw video data            from the video sources 522, and preparing the received video            data for event processing and long-term storage in the video            storage database 514;        -   Camera control module 618 for generating and sending            server-initiated control commands to modify the operation            modes of the video sources, and/or receiving and forwarding            user-initiated control commands to modify the operation            modes of the video sources 522;        -   Event detection module 620 for detecting motion event            candidates in video streams from each of the video sources            522, including motion track identification, false positive            suppression, and event mask generation and caching;        -   Event categorization module 622 for categorizing motion            events detected in received video streams;        -   Zone creation module 624 for generating zones of interest in            accordance with user input;        -   Person identification module 626 for identifying            characteristics associated with presence of humans in the            received video streams;        -   Filter application module 628 for selecting event filters            (e.g., event categories, zones of interest, a human filter,            etc.) and applying the selected event filter to past and new            motion events detected in the video streams;        -   Zone monitoring module 630 for monitoring motions within            selected zones of interest and generating notifications for            new motion events detected within the selected zones of            interest, where the zone monitoring takes into account            changes in surrounding context of the zones and is not            confined within the selected zones of interest;        -   Real-time motion event presentation module 632 for            dynamically changing characteristics of event indicators            displayed in user interfaces as new event filters, such as            new event categories or new zones of interest, are created,            and for providing real-time notifications as new motion            events are detected in the video streams; and        -   Event post-processing module 634 for providing summary            time-lapse for past motion events detected in video streams,            and providing event and category editing functions to user            for revising past event categorization results; and    -   server data 636 storing data for use in data processing for        motion event monitoring and review, including but not limited        to:        -   Video storage database 514 storing raw video data associated            with each of the video sources 522 (each including one or            more cameras 118) of each reviewer account, as well as event            categorization models (e.g., event clusters, categorization            criteria, etc.), event categorization results (e.g.,            recognized event categories, and assignment of past motion            events to the recognized event categories, representative            events for each recognized event category, etc.), event            masks for past motion events, video segments for each past            motion event, preview video (e.g., sprites) of past motion            events, and other relevant metadata (e.g., names of event            categories, location of the cameras 118, creation time,            duration, DTPZ settings of the cameras 118, etc.) associated            with the motion events; and        -   Account database 516 for storing account information for            reviewer accounts, including login-credentials, associated            video sources, relevant user and hardware characteristics            (e.g., service tier, camera model, storage capacity,            processing capabilities, etc.), user interface settings,            monitoring preferences, etc.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various implementations. In some implementations, thememory 606, optionally, stores a subset of the modules and datastructures identified above. Furthermore, the memory 606, optionally,stores additional modules and data structures not described above.

FIG. 7 is a block diagram illustrating a representative client device504 associated with a reviewer account in accordance with someimplementations. The client device 504, typically, includes one or moreprocessing units (CPUs) 702, one or more network interfaces 704, memory706, and one or more communication buses 708 for interconnecting thesecomponents (sometimes called a chipset). The client device 504 alsoincludes a user interface 710. The user interface 710 includes one ormore output devices 712 that enable presentation of media content,including one or more speakers and/or one or more visual displays. Theuser interface 710 also includes one or more input devices 714,including user interface components that facilitate user input such as akeyboard, a mouse, a voice-command input unit or microphone, a touchscreen display, a touch-sensitive input pad, a gesture capturing camera,or other input buttons or controls. Furthermore, the client device 504optionally uses a microphone and voice recognition or a camera andgesture recognition to supplement or replace the keyboard. In someimplementations, the client device 504 includes one or more cameras,scanners, or photo sensor units for capturing images. In someimplementations, the client device 504 optionally includes a locationdetection device 715, such as a GPS (global positioning satellite) orother geo-location receiver, for determining the location of the clientdevice 504.

The memory 706 includes high-speed random access memory, such as DRAM,SRAM, DDR RAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. The memory 706, optionally, includes one or morestorage devices remotely located from the one or more processing units702. The memory 706, or alternatively the non-volatile memory within thememory 706, includes a non-transitory computer readable storage medium.In some implementations, the memory 706, or the non-transitory computerreadable storage medium of memory 706, stores the following programs,modules, and data structures, or a subset or superset thereof:

-   -   Operating system 716 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   Network communication module 718 for connecting the client        device 504 to other computing devices (e.g., the video server        system 508 and the video sources 522) connected to the one or        more networks 162 via the one or more network interfaces 704        (wired or wireless);    -   Presentation module 720 for enabling presentation of information        (e.g., user interfaces for application(s) 726 or the client-side        module 502, widgets, web sites and web pages thereof, and/or        games, audio and/or video content, text, etc.) at the client        device 504 via the one or more output devices 712 (e.g.,        displays, speakers, etc.) associated with the user interface        710;    -   Input processing module 722 for detecting one or more user        inputs or interactions from one of the one or more input devices        714 and interpreting the detected input or interaction;    -   Web browser module 724 for navigating, requesting (e.g., via        HTTP), and displaying websites and web pages thereof, including        a web interface for logging into a reviewer account, controlling        the video sources associated with the reviewer account,        establishing and selecting event filters, and editing and        reviewing motion events detected in the video streams of the        video sources;    -   One or more applications 726 for execution by the client device        504 (e.g., games, social network applications, smart home        applications, and/or other web or non-web based applications);    -   Client-side module 502, which provides client-side data        processing and functionalities for monitoring and reviewing        motion events detected in the video streams of one or more video        sources, including but not limited to:        -   Account registration module 728 for establishing a reviewer            account and registering one or more video sources with the            video server system 508;        -   Camera setup module 730 for setting up one or more video            sources within a local area network, and enabling the one or            more video sources to access the video server system 508 on            the Internet through the local area network;        -   Camera control module 732 for generating control commands            for modifying an operating mode of the one or more video            sources in accordance with user input;        -   Event review interface module 734 for providing user            interfaces for reviewing event timelines, editing event            categorization results, selecting event filters, presenting            real-time filtered motion events based on existing and newly            created event filters (e.g., event categories, zones of            interest, a human filter, etc.), presenting real-time            notifications (e.g., pop-ups) for newly detected motion            events, and presenting smart time-lapse of selected motion            events;        -   Zone creation module 736 for providing a user interface for            creating zones of interest for each video stream in            accordance with user input, and sending the definitions of            the zones of interest to the video server system 508; and        -   Notification module 738 for generating real-time            notifications for all or selected motion events on the            client device 504 outside of the event review user            interface; and    -   client data 770 storing data associated with the reviewer        account and the video sources 522, including, but is not limited        to:        -   Account data 772 storing information related with the            reviewer account, and the video sources, such as cached            login credentials, camera characteristics, user interface            settings, display preferences, etc.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, modules or datastructures, and thus various subsets of these modules may be combined orotherwise re-arranged in various implementations. In someimplementations, memory 706, optionally, stores a subset of the modulesand data structures identified above. Furthermore, the memory 706,optionally, stores additional modules and data structures not describedabove.

In some implementations, at least some of the functions of the videoserver system 508 are performed by the client device 504, and thecorresponding sub-modules of these functions may be located within theclient device 504 rather than the video server system 508. In someimplementations, at least some of the functions of the client device 504are performed by the video server system 508, and the correspondingsub-modules of these functions may be located within the video serversystem 508 rather than the client device 504. The client device 504 andthe video server system 508 shown in FIGS. 6-7, respectively, are merelyillustrative, and different configurations of the modules forimplementing the functions described herein are possible in variousimplementations.

FIG. 8 is a block diagram illustrating a representative camera 118 inaccordance with some implementations. In some implementations, thecamera 118 includes one or more processing units (e.g., CPUs, ASICs,FPGAs, microprocessors, and the like) 802, one or more communicationinterfaces 804, memory 806, and one or more communication buses 808 forinterconnecting these components (sometimes called a chipset). In someimplementations, the camera 118 includes one or more input devices 810such as one or more buttons for receiving input and one or moremicrophones. In some implementations, the camera 118 includes one ormore output devices 812 such as one or more indicator lights, a soundcard, a speaker, a small display for displaying textual information anderror codes, etc. In some implementations, the camera 118 optionallyincludes a location detection device 814, such as a GPS (globalpositioning satellite) or other geo-location receiver, for determiningthe location of the camera 118.

The memory 806 includes high-speed random access memory, such as DRAM,SRAM, DDR RAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. The memory 806, or alternatively the non-volatilememory within the memory 806, includes a non-transitory computerreadable storage medium. In some implementations, the memory 806, or thenon-transitory computer readable storage medium of the memory 806,stores the following programs, modules, and data structures, or a subsetor superset thereof:

-   -   Operating system 816 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   Network communication module 818 for connecting the camera 118        to other computing devices (e.g., the video server system 508,        the client device 504, network routing devices, one or more        controller devices, and networked storage devices) connected to        the one or more networks 162 via the one or more communication        interfaces 804 (wired or wireless);    -   Video control module 820 for modifying the operation mode (e.g.,        zoom level, resolution, frame rate, recording and playback        volume, lighting adjustment, AE and IR modes, etc.) of the        camera 118, enabling/disabling the audio and/or video recording        functions of the camera 118, changing the pan and tilt angles of        the camera 118, resetting the camera 118, and/or the like;    -   Video capturing module 824 for capturing and generating a video        stream and sending the video stream to the video server system        508 as a continuous feed or in short bursts;    -   Video caching module 826 for storing some or all captured video        data locally at one or more local storage devices (e.g., memory,        flash drives, internal hard disks, portable disks, etc.);    -   Local video processing module 828 for performing preliminary        processing of the captured video data locally at the camera 118,        including for example, compressing and encrypting the captured        video data for network transmission, preliminary motion event        detection, preliminary false positive suppression for motion        event detection, preliminary motion vector generation, etc.; and    -   Camera data 830 storing data, including but not limited to:        -   Camera settings 832, including network settings, camera            operation settings, camera storage settings, etc.; and        -   Video data 834, including video segments and motion vectors            for detected motion event candidates to be sent to the video            server system 508.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various implementations. In some implementations, thememory 806, optionally, stores a subset of the modules and datastructures identified above. Furthermore, memory 806, optionally, storesadditional modules and data structures not described above.

User Interfaces for Video Monitoring

Attention is now directed towards implementations of user interfaces andassociated processes that may be implemented on a respective clientdevice 504 with one or more speakers enabled to output sound, zero ormore microphones enabled to receive sound input, and a touch screen 906enabled to receive one or more contacts and display information (e.g.,media content, webpages and/or user interfaces for an application).FIGS. 9A-9BB illustrate example user interfaces for monitoring andfacilitating review of motion events in accordance with someimplementations.

Although some of the examples that follow will be given with referenceto inputs on touch screen 906 (where the touch sensitive surface and thedisplay are combined), in some implementations, the device detectsinputs on a touch-sensitive surface that is separate from the display.In some implementations, the touch sensitive surface has a primary axisthat corresponds to a primary axis on the display. In accordance withthese implementations, the device detects contacts with thetouch-sensitive surface at locations that correspond to respectivelocations on the display. In this way, user inputs detected by thedevice on the touch-sensitive surface are used by the device tomanipulate the user interface on the display of the device when thetouch-sensitive surface is separate from the display. It should beunderstood that similar methods are, optionally, used for other userinterfaces described herein.

Additionally, while the following examples are given primarily withreference to finger inputs (e.g., finger contacts, finger tap gestures,finger swipe gestures, etc.), it should be understood that, in someimplementations, one or more of the finger inputs are replaced withinput from another input device (e.g., a mouse based input or stylusinput). For example, a swipe gesture is, optionally, replaced with amouse click (e.g., instead of a contact) followed by movement of thecursor along the path of the swipe (e.g., instead of movement of thecontact). As another example, a tap gesture is, optionally, replacedwith a mouse click while the cursor is located over the location of thetap gesture (e.g., instead of detection of the contact followed byceasing to detect the contact). Similarly, when multiple user inputs aresimultaneously detected, it should be understood that multiple computermice are, optionally, used simultaneously, or a mouse and fingercontacts are, optionally, used simultaneously.

FIGS. 9A-9BB show user interface 908 displayed on client device 504(e.g., a tablet, laptop, mobile phone, or the like); however, oneskilled in the art will appreciate that the user interfaces shown inFIGS. 9A-9BB may be implemented on other similar computing devices. Theuser interfaces in FIGS. 9A-9BB are used to illustrate the processesdescribed herein, including the processes and/or methods described withrespect to FIGS. 10, 12A-12B, 13A-13B, 14A-14B, 15A-15C, and 16A-16B.

For example, the client device 504 is the portable electronic device 166(FIG. 1) such as a laptop, tablet, or mobile phone. Continuing with thisexample, the user of the client device 504 (sometimes also herein calleda “reviewer”) executes an application (e.g., the client-side module 502,FIGS. 5 and 7) used to monitor and control the smart home environment100 and logs into a user account registered with the smart home providersystem 164 or a component thereof (e.g., the video server system 508,FIGS. 5-6). In this example, the smart home environment 100 includes theone or more cameras 118, whereby the user of the client device 504 isable to control, review, and monitor video feeds from the one or morecameras 118 with the user interfaces for the application displayed onthe client device 504 shown in FIGS. 9A-9BB.

FIG. 9A illustrates the client device 504 displaying a firstimplementation of a video monitoring user interface (UI) of theapplication on the touch screen 906. In FIG. 9A, the video monitoring UIincludes three distinct regions: a first region 903, a second region905, and a third region 907. In FIG. 9A, the first region 903 includes avideo feed from a respective camera among the one or more camera 118associated with the smart home environment 100. For example, therespective camera is located on the back porch of the user's domicile orpointed out of a window of the user's domicile. The first region 903includes the time 911 of the video feed being displayed in the firstregion 903 and also an indicator 912 indicating that the video feedbeing displayed in the first region 903 is a live video feed.

In FIG. 9A, the second region 905 includes an event timeline 910 and acurrent video feed indicator 909 indicating the temporal position of thevideo feed displayed in the first region 903 (i.e., the point ofplayback for the video feed displayed in the first region 903). In FIG.9A, the video feed displayed in the first region 903 is a live videofeed from the respective camera. In some implementations, the video feeddisplayed in the first region 903 may be previously recorded videofootage. For example, the user of the client device 504 may drag theindicator 909 to any position on the event timeline 910 causing theclient device 504 to display the video feed from that point in timeforward in the first region 903. In another example, the user of theclient device 504 may perform a substantially horizontal swipe gestureon the event timeline 910 to scrub between points of the recorded videofootage causing the indicator 909 to move on the event timeline 910 andalso causing the client device 504 to display the video feed from thatpoint in time forward in the first region 903.

The second region 905 also includes affordances 913 for changing thescale of the event timeline 910: 5 minute affordance 913A for changingthe scale of the event timeline 910 to 5 minutes, 1 hour affordance 913Bfor changing the scale of the event timeline 910 to 1 hour, andaffordance 24 hours 913C for changing the scale of the event timeline910 to 24 hours. In FIG. 9A, the scale of the event timeline 910 is 1hour as evinced by the darkened border surrounding the 1 hour affordance913B and also the temporal tick marks shown on the event timeline 910.The second region 905 also includes affordances 914 for changing thedate associated with the event timeline 910 to any day within thepreceding week: Monday affordance 914A, Tuesday affordance 914B,Wednesday affordance 914C, Thursday affordance 914D, Friday affordance914E, Saturday affordance 914F, Sunday affordance 914G, and Todayaffordance 914H. In FIG. 9A, the event timeline 910 is associated withthe video feed from today as evinced by the darkened border surroundingToday affordance 914H. In some implementations, an affordance is a userinterface element that is user selectable or manipulatable on agraphical user interface.

In FIG. 9A, the second region 905 further includes: “Make Time-Lapse”affordance 915, which, when activated (e.g., via a tap gesture), enablesthe user of the client device 504 to select a portion of the eventtimeline 910 for generation of a time-lapse video clip (as shown inFIGS. 9N-9Q); “Make Clip” affordance 916, which, when activated (e.g.,via a tap gesture), enables the user of the client device 504 to selecta motion event or a portion of the event timeline 910 to save as a videoclip; and “Make Zone” affordance 917, which, when activated (e.g., via atap gesture), enables the user of the client device 504 to create a zoneof interest on the current field of view of the respective camera (asshown in FIGS. 9K-9M). In some embodiments, the time-lapse video clipand saved non-time-lapse video clips are associated with the useraccount of the user of the client device 504 and stored by the servervideo server system 508 (e.g., in the video storage database 516, FIGS.5-6). In some embodiments, the user of the client device 504 is able toaccess his/her saved time-lapse video clip and saved non-time-lapsevideo clips by entering the login credentials for his/her for useraccount.

In FIG. 9A, the video monitoring UI also includes a third region 907with a list of categories with recognized event categories and createdzones of interest. FIG. 9A also illustrates the client device 504detecting a contact 918 (e.g., a tap gesture) at a locationcorresponding to the first region 903 on the touch screen 906.

FIG. 9B illustrates the client device 504 displaying additional videocontrols in response to detecting the contact 918 in FIG. 9A. In FIG.9B, the first region 903 of the video monitoring UI includes: anelevator bar with a handle 919 for adjusting the zoom magnification ofthe video feed displayed in the first region 903, affordance 920A forreducing the zoom magnification of the video feed, and affordance 920Bfor increasing the zoom magnification of the video feed. In FIG. 9B, thefirst region 903 of the video monitoring UI also includes: affordance921A for enabling/disabling the microphone of the respective cameraassociated with the video feed; affordance 921B for rewinding the videofeed by 30 seconds; affordance 921C for pausing the video feed displayedin the first region 903; affordance 921D for adjusting the playbackvolume of the video feed; and affordance 921E for displaying the videofeed in full screen mode.

FIG. 9C illustrates the client device 504 displaying the event timeline910 in the second region 905 with event indicators 922A, 922B, 922C,922D, 922E, and 922F corresponding to detected motion events. In someimplementations, the location of a respective event indicator 922 on theevent timeline 910 corresponds to the time at which a motion eventcorrelated with the respective event indicator 922 was detected. Thedetected motion events correlated with the event indicators 922A, 922B,922C, 922D, 922E, and 922F are uncategorized motion events as no eventcategories have been recognized by the video server system 508 and nozones of interest have been created by the user of the client device504. In some implementations, for example, the list of categories in thethird region 907 includes an entry for uncategorized motion events(e.g., the motion events correlated with event indicators 922A, 922B,922C, 922D, 922E, and 922F) with a filter affordance forenabling/disabling display of event indicators for the uncategorizedmotion events on the event timeline 910.

FIG. 9D illustrates the client device 504 displaying the event timeline910 in the second region 905 with additional event indicators 922G,922H, 922I, and 922J. In FIG. 9D, the list of categories in the thirdregion 907 includes an entry 924A for newly recognized event category A.The entry 924A for recognized event category A includes: a displaycharacteristic indicator 925A representing the display characteristicfor event indicators corresponding to motion events assigned to eventcategory A (e.g., vertical stripes); an indicator filter 926A forenabling/disabling display of event indicators on the event timeline 910for motion events assigned to event category A; and a notificationsindicator 927A for enabling/disabling notifications sent in response todetection of motion events assigned to event category A. In FIG. 9D,display of event indicators for motion events corresponding to eventcategory A is enabled as evinced by the check mark corresponding toindicator filter 926A and notifications are enabled.

In FIG. 9D, motion events correlated with the event indicators 922A,922C, 922D, and 922E have been retroactively assigned to event categoryA as shown by the changed display characteristic of the event indicators922A, 922C, 922D, and 922E (e.g., vertical stripes). In someimplementations, the display characteristic is a fill color of the eventindicator, a shading pattern of the event indicator, an icon overlaid onthe event indicator, or the like. In some implementations, thenotifications are messages sent by the video server system 508 (FIGS.5-6) via email to an email address linked to the user's account or via aSMS or voice call to a phone number linked to the user's account. Insome implementations, the notifications are audible tones or vibrationsprovided by the client device 504.

FIG. 9E illustrates the client device 504 displaying an entry 924B fornewly recognized event category B in the list of categories in the thirdregion 907. The entry 924B for recognized event category B includes: adisplay characteristic indicator 925B representing the displaycharacteristic for event indicators corresponding to motion eventsassigned to event category B (e.g., a diagonal shading pattern); anindicator filter 926B for enabling/disabling display of event indicatorson the event timeline 910 for motion events assigned to event categoryB; and a notifications indicator 927B for enabling/disablingnotifications sent in response to detection of motion events assigned toevent category B. In FIG. 9E, display of event indicators for motionevents corresponding to event category B is enabled as evinced by thecheck mark corresponding to indicator filter 926B and notifications areenabled. In FIG. 9E, motion events correlated with the event indicators922F, 922G, 922H, 922J, and 922K have been retroactively assigned toevent category B as shown by the changed display characteristic of theevent indicators 922F, 922G, 922H, 922J, and 922K (e.g., the diagonalshading pattern).

FIG. 9E also illustrates client device 504 displaying a notification 928for a newly detected respective motion event corresponding to eventindicator 922L. For example, event category B is recognized prior to orconcurrent with detecting the respective motion event. For example, asthe respective motion event is detected and assigned to event categoryB, an event indicator 922L is displayed on the event timeline 910 withthe display characteristic for event category B (e.g., the diagonalshading pattern). Continuing with this example, after or as the eventindicator 922L is displayed on the event timeline 910, the notification928 pops-up from the event indicator 922L. In FIG. 9E, the notification928 notifies the user of the client device 504 that the motion eventdetected at 12:32:52 pm was assigned to event category B. In someimplementations, the notification 928 is at least partially overlaid onthe video feed displayed in the first region 903. In someimplementations, the notification 928 pops-up from the event timeline910 and is at least partially overlaid on the video feed displayed inthe first region 903 (e.g., in the center of the first region 903 or atthe top of the first region 903 as a banner notification). FIG. 9E alsoillustrates the client device 504 detecting a contact 929 (e.g., a tapgesture) at a location corresponding to the notifications indicator 927Aon the touch screen 906.

FIG. 9F shows the notifications indicator 927A in the third region 907as disabled, shown by the line through the notifications indicator 927A,in response to detecting the contact 929 in FIG. 9E. FIG. 9F illustratesthe client device 504 detecting a contact 930 (e.g., a tap gesture) at alocation corresponding to the indicator filter 926A on the touch screen906.

FIG. 9G shows the indicator filter 926A as unchecked in response todetecting the contact 930 in FIG. 9F. Moreover, in FIG. 9G, the clientdevice 504 ceases to display the event indicators 922A, 922C, 922D, and922E, which correspond to motion events assigned to event category A, onthe event timeline 910 in response to detecting the contact 930 in FIG.9F. FIG. 9G also illustrates the client device 504 detecting a contact931 (e.g., a tap gesture) at a location corresponding to event indicator922B on the touch screen 906.

FIG. 9H illustrates the client device 504 displaying a dialog box 923for a respective motion event correlated with the event indicator 922Bin response to detecting selection of the event indicator 922B in FIG.9G. In some implementations, the dialog box 923 may be displayed inresponse to sliding or hovering over the event indicator 922B. In FIG.9H, the dialog box 923 includes the time the respective motion event wasdetected (e.g., 11:37:40 am) and a preview 932 of the respective motionevent (e.g., a static image, a series of images, or a video clip). InFIG. 9H, the dialog box 923 also includes an affordance 933, which, whenactivated (e.g., with a tap gesture), causes the client device 504 todisplay an editing user interface (UI) for the event category to whichthe respective motion event is assigned (if any) and/or the zone orinterest which the respective motion event touches or overlaps (if any).FIG. 9H also illustrates the client device 504 detecting a contact 934(e.g., a tap gesture) at a location corresponding to the entry 924B forevent category B on the touch screen 906.

FIG. 9I illustrates the client device 504 displaying an editing userinterface (UI) for event category B in response to detecting selectionof the entry 924B in FIG. 9H. In FIG. 9I, the editing UI for eventcategory B includes two distinct regions: a first region 935; and asecond region 937. The first region 935 includes representations 936(sometimes also herein called “sprites”) of motion events assigned toevent category B, where a representation 936A corresponds to the motionevent correlated with the event indicator 922F, a representation 936Bcorresponds to the motion event correlated with the event indicator922G, a representation 936C corresponds to the motion event correlatedwith the event indicator 922L, a representation 936D corresponds to themotion event correlated with the event indicator 922K, and arepresentation 936E corresponds to the motion event correlated with theevent indicator 922J. In some implementations, each of therepresentations 936 is a series of frames or a video clip of arespective motion event assigned to event category B. For example, inFIG. 9I, each of the representations 936 corresponds to a motion eventof a bird flying from left to right across the field of view of therespective camera. In FIG. 9I, each of the representations 936 isassociated with a checkbox 941. In some implementations, when arespective checkbox 941 is unchecked (e.g., with a tap gesture) themotion event corresponding to the respective checkbox 941 is removedfrom the event category B and, in some circumstances, the event categoryB is re-computed based on the removed motion event. For example, thecheckboxes 941 enable the user of the client device 504 to remove motionevents incorrectly assigned to an event category so that similar motionevents are not assigned to the event category in the future.

In FIG. 9I, the first region 935 further includes: a save/exitaffordance 938 for saving changes made to event category B or exitingthe editing UI for event category B; a label text entry box 939 forrenaming the label for the event category from the default name (“eventcategory B”) to a custom name; and a notifications indicator 940 forenabling/disabling notifications sent in response to detection of motionevents assigned to event category B. In FIG. 9I, the second region 937includes a representation of the video feed from the respective camerawith a linear motion vector 942 representing the typical path of motionfor motion events assigned event category B. In some implementations,the representation of the video feed is a static image recently capturedfrom the video feed or the live video feed. FIG. 9I also illustrates theclient device 504 detecting a contact 943 (e.g., a tap gesture) at alocation corresponding to the checkbox 941C on the touch screen 906 anda contact 944 (e.g., a tap gesture) at a location corresponding to thecheckbox 941E on the touch screen 906. For example, the user of theclient device 504 intends to remove the motion events corresponding tothe representations 936C and 936E as neither shows a bird flying in awest to northeast direction.

FIG. 9J shows the checkbox 941C corresponding to the motion eventcorrelated with the event indicator 922L and the checkbox 941Ecorresponding to the motion event correlated with the event indicator922J as unchecked in response to detecting the contact 943 and thecontact 944, respectively, in FIG. 9I. FIG. 9J also shows the label forthe event category as “Birds in Flight” in the label text entry box 939as opposed to “event category B” in FIG. 9I. FIG. 9J illustrates theclient device 504 detecting a contact 945 (e.g., a tap gesture) at alocation corresponding to the save/exit affordance 938 on the touchscreen 906. For example, in response to detecting the contact 945, theclient device 504 sends a message to the video server system 508indicating removal of the motion events corresponding to therepresentations 936C and 936E from event category B so as to re-computethe algorithm for assigning motion events to event category B (nowrenamed “Birds in Flight”).

FIG. 9K illustrates the client device 504 displaying event indicators922J and 922L with a changed display characteristic corresponding touncategorized motion events (i.e., no fill) in response to removal ofthe representations 936C and 936E, which correspond to the motion eventscorrelated with the event indicators 922J and 922L, from event categoryB in FIGS. 9I-9J. FIG. 9K also illustrates the client device 504displaying “Birds in Flight” as the label for the entry 924B in the listof categories in the third region 907 in response to the changed labelentered in FIG. 9J. FIG. 9K further illustrates the client device 504detecting a contact 946 (e.g., a tap gesture) at a locationcorresponding to “Make Zone” affordance 917 on the touch screen 906.

FIG. 9L illustrates the client device 504 displaying a customizableoutline 947A for a zone of interest on the touch screen 906 in responseto detecting selection of the “Make Zone” affordance 917 in FIG. 9K. InFIG. 9L, the customizable outline is rectangular, however, one of skillin the art will appreciate that the customizable outline may bepolyhedral, circular, any other shape, or a free hand shape drawn on thetouch screen 906 by the user of the client device 504. In someimplementations, the customizable outline 947A may be adjusted byperforming a dragging gesture with any corner or side of thecustomizable outline 947A. FIG. 9L also illustrates the client device504 detecting a dragging gesture whereby contact 949 is moved from afirst location 950A corresponding to the right side of the customizableoutline 947A to a second location 950B. In FIG. 9L, the first region 903includes “Save Zone” affordance 952, which, when activated (e.g., with atap gesture), causes creation of the zone of interest corresponding tothe customizable outline 947.

FIG. 9M illustrates the client device 504 displaying an expandedcustomizable outline 947B on the touch screen 906 in response todetecting the dragging gesture in FIG. 9L. FIG. 9M also illustrates theclient device 504 detecting a contact 953 (e.g., a tap gesture) at alocation corresponding to the “Save Zone” affordance 952 on the touchscreen 906. For example, in response to detecting selection of the “SaveZone” affordance 952, the client device 504 causes creation of the zoneof interest corresponding to the expanded customizable outline 947B bysending a message to the video server system 508 indicating thecoordinates of the expanded customizable outline 947B.

FIG. 9N illustrates the client device 504 displaying an entry 924C fornewly created zone A in the list of categories in the third region 907in response to creating the zone of interest in FIGS. 9L-9M. The entry924C for newly created zone A includes: a display characteristicindicator 925C representing the display characteristic for eventindicators corresponding to motion events that touch or overlap zone A(e.g., an ‘X’ at the bottom of the event indicator); an indicator filter926C for enabling/disabling display of event indicators on the eventtimeline 910 for motion events that touch or overlap zone A; and anotifications indicator 927C for enabling/disabling notifications sentin response to detection of motion events that touch or overlap zone A.In FIG. 9N, display of event indicators for motion events that touch oroverlap zone A is enabled as evinced by the check mark corresponding toindicator filter 926C and notifications are enabled. In FIG. 9N, themotion event correlated with the event indicator 922M has beenretroactively associated with zone A as shown by the changed displaycharacteristic of the event indicator 922M (e.g., the ‘X’ at the bottomof the event indicator 922M). FIG. 9N also illustrates the client device504 detecting a contact 954 (e.g., a tap gesture) at a locationcorresponding to the “Make Time-Lapse” affordance 915 on the touchscreen 906.

FIG. 9O illustrates the client device 504 displaying controls forgenerating a time-lapse video clip in response to detecting selection ofthe “Make Time-Lapse” affordance 915 in FIG. 9N. In FIG. 9O, the secondregion 905 includes a start time entry box 956A for entering/changing astart time of the time-lapse video clip to be generated and an end timeentry box 956B for entering/changing an end time of the time-lapse videoclip to be generated. In FIG. 9O, the second region 905 also includes astart time indicator 957A and an end time indicator 957B on the eventtimeline 910, which indicate the start and end times of the time-lapsevideo clip to be generated. In some implementations, the locations ofthe start time indicator 957A and the end time indicator 957B may bemoved on the event timeline 910 via pulling/dragging gestures.

In FIG. 9O, the second region 905 further includes a “Create Time-lapse”affordance 958, which, when activated (e.g., with a tap gesture) causesgeneration of the time-lapse video clip based on the selected portion ofthe event timeline 910 corresponding to the start and end timesdisplayed by the start time entry box 956A (e.g., 12:20:00 pm) and theend time entry box 956B (e.g., 12:42:30 pm) and also indicated by thestart time indicator 957A and the end time indicator 957B. In someimplementations, prior to generation of the time-lapse video clip andafter selection of the “Create Time-Lapse” affordance 958, the clientdevice 504 displays a dialog box that enables the user of the clientdevice 504 to select a length of the time-lapse video clip (e.g., 30,60, 90, etc. seconds). In FIG. 9O, the second region 905 furtherincludes an “Abort” affordance 959, which, when activated (e.g., with atap gesture) causes the client device 504 to display a previous UI(e.g., the video monitoring UI in FIG. 9N). FIG. 9O further illustratesthe client device 504 detecting a contact 955 (e.g., a tap gesture) at alocation corresponding to the “Create Time-Lapse” affordance 958 on thetouch screen 906.

In some implementations, the time-lapse video clip is generated by theclient device 504, the video server system 508, or a combinationthereof. In some implementations, motion events within the selectedportion of the event timeline 910 are played at a slower speed than thebalance of the selected portion of the event timeline 910. In someimplementations, motion events within the selected portion of the eventtimeline 910 that are assigned to enabled event categories and motionevents within the selected portion of the event timeline 910 that touchor overlap enabled zones are played at a slower speed than the balanceof the selected portion of the event timeline 910 including motionevents assigned to disabled event categories and motion events thattouch or overlap disabled zones.

FIG. 9P illustrates the client device 504 displaying a notification 961overlaid on the first region 903 in response to detecting selection ofthe “Create Time-Lapse” affordance 958 in FIG. 9O. In FIG. 9P, thenotification 961 indicates that the time-lapse video clip is beingprocessed and also includes an exit affordance 962, which, whenactivated (e.g., with a tap gesture), causes the client device 504 theclient device 504 to dismiss the notification 961. At a time subsequent,the notification 961 in FIG. 9Q indicates that processing of thetime-lapse video clip is complete and includes a “Play Time-Lapse”affordance 963, which, when activated (e.g., with a tap gesture), causesthe client device 504 to play the time-lapse video clip. FIG. 9Qillustrates the client device 504 detecting a contact 964 at a locationcorresponding to the exit affordance 962 on the touch screen 906.

FIG. 9R illustrates the client device 504 ceasing to display thenotification 961 in response to detecting selection of the exitaffordance 962 in FIG. 9Q. FIG. 9R also illustrates the client device504 detecting a pinch-in gesture with contacts 965A and 965B relative toa respective portion of the video feed in the first region 903 on thetouch screen 906.

FIG. 9S illustrates the client device 504 displaying a zoomed-in portionof the video feed in response to detecting the pinch-in gesture on thetouch screen 906 in FIG. 9R. In some implementations, the zoomed-inportion of the video feed corresponds to a software-based zoom performedlocally by the client device 504 on the respective portion of the videofeed corresponding to the pinch-in gesture in FIG. 9R. In FIG. 9S, thehandle 919 of the elevator bar indicates the current zoom magnificationof the video feed and a perspective box 969 indicates the zoomed-inportion 970 relative to the full field of view of the respective camera.In some implementations, the video monitoring UI further indicates thecurrent zoom magnification in text.

In FIG. 9S, the video controls in the first region 903 further includean enhancement affordance 968, which, when activated (e.g., with a tapgesture) causes the client device 504 to send a zoom command to therespective camera. In some implementations, the zoom command causes therespective camera to perform a zoom operation at the zoom magnificationcorresponding to the distance between contacts 965A and 965B of thepinch-in gesture in FIG. 9R on the respective portion of the video feedcorresponding to the pinch-in gesture in FIG. 9R. In someimplementations, the zoom command is relayed to the respective camera bythe video server system 508. In some implementations, the zoom commandis sent directly to the respective camera by the client device 504. FIG.9S also illustrates the client device 504 detecting a contact 967 at alocation corresponding to the enhancement affordance 968 on the touchscreen 906.

FIG. 9T illustrates the client device 504 displaying a dialog box 971 inresponse to detecting selection of the enhancement affordance 968 inFIG. 9S. In FIG. 9T, the dialog box 971 warns the user of the clientdevice 504 that enhancement of the video feed will cause changes to therecorded video footage and also causes changes to any previously createdzones of interest. In FIG. 9T, the dialog box 971 includes: a cancelaffordance 972, which, when activated (e.g., with a tap gesture) causesthe client device 504 to cancel of the enhancement operation andconsequently cancel sending of the zoom command; and an enhanceaffordance 973, when activated (e.g., with a tap gesture) causes theclient device 504 to send the zoom command to the respective camera.FIG. 9T also illustrates the client device 504 detecting a contact 974at a location corresponding to the enhance affordance 973 on the touchscreen 906.

FIG. 9U illustrates the client device 504 displaying the zoomed-inportion of the video feed at a higher resolution as compared to FIG. 9Sin response to detecting selection of the enhance affordance 973 in FIG.9T. In some implementations, in response to sending the zoom command,the client device 504 receives a higher resolution video feed (e.g.,780i, 720p, 1080i, or 1080p) of the zoomed-in portion of the video feed.In FIG. 9U, the video controls in the first region 903 further include azoom reset affordance 975, which, when activated (e.g., with a tapgesture) causes the client device 504 reset the zoom magnification ofthe video feed to its original setting (e.g., as in FIG. 9R prior to thepinch-in gesture). FIG. 9U also illustrates the client device 504detecting a contact 978 at a location corresponding to the 24 hoursaffordance 913C on the touch screen 906.

FIG. 9V illustrates the client device 504 displaying the event timeline910 with a 24 hour scale in response to detecting selection of the 24hours affordance 913C in FIG. 9U. FIG. 9V also illustrates the clientdevice 504 detecting a contact 980 (e.g., a tap gesture) at a locationcorresponding to an event indicator 979 on the touch screen 906.

FIG. 9W illustrates the client device 504 displaying a dialog box 981for respective motion events correlated with the event indicator 979 inresponse to detecting selection of the event indicator 979 in FIG. 9V.In some implementations, the dialog box 981 may be displayed in responseto sliding or hovering over the event indicator 979. In FIG. 9W, thedialog box 981 includes the times at which the respective motion eventswere detected (e.g., 6:35:05 am, 6:45:15 am, and 6:52:45 am). In FIG.9W, the dialog box 981 also includes previews 982A, 982B, and 982C ofthe respective motion events (e.g., a static image, a series of images,or a video clip).

FIG. 9X illustrates the client device 504 displaying a secondimplementation of a video monitoring user interface (UI) of theapplication on the touch screen 906. In FIG. 9X, the video monitoring UIincludes two distinct regions: a first region 986; and a second region988. In FIG. 9X, the first region 986 includes a video feed from arespective camera among the one or more camera 118 associated with thesmart home environment 100. For example, the respective camera islocated on the back porch of the user's domicile or pointed out of awindow of the user's domicile. The first region 986 includes anindicator 990 indicating that the video feed being displayed in thefirst region 986 is a live video feed. In some implementations, if thevideo feed being displayed in the first region 986 is recorded videofootage, the indicator 990 is instead displayed as a “Go Live”affordance, which, when activated (e.g., with a tap gesture), causes theclient device to display the live video feed from the respective camerain the first region 986.

In FIG. 9X, the second region 988 includes a text box 993 indicating thetime and date of the video feed being displayed in the first region 986.In FIG. 9X, the second region 988 also includes: an affordance 991 forrewinding the video feed displayed in the first region 986 by 30seconds; and an affordance 992 for enabling/disabling the microphone ofthe respective camera associated with the video feed displayed in thefirst region 986. In FIG. 9X, the second region 988 further includes a“Motion Events Feed” affordance 994, which, when activated (e.g., via atap gesture), causes the client device 504 to display a motion eventtimeline (e.g., the user interface shown in FIGS. 9Y-9Z). FIG. 9X alsoillustrates the client device 504 detecting a contact 996 (e.g., a tapgesture) at a location corresponding to the “Motion Events Feed”affordance 994 on the touch screen 906.

FIG. 9Y illustrates the client device 504 displaying a first portion ofa motion events feed 997 in response to detecting selection of the“Motion Events Feed” affordance 994 in FIG. 9X. In FIG. 9Y, the motionevents feed 997 includes representations 998 (sometimes also hereincalled “sprites”) of motion events. In FIG. 9Y, each of therepresentations 998 is associated with a time at which the motion eventwas detected, and each of the representations 998 is associated with anevent category to which it is assigned to the motion event (if any)and/or a zone which it touches or overlaps (if any). In FIG. 9Y, each ofthe representations 998 is associated with a unique displaycharacteristic indicator 925 representing the display characteristic forthe event category to which it is assigned (if any) and/or the zonewhich it touches or overlaps (if any). For example, the representation998A corresponds to a respective motion event that was detected at12:39:45 pm which touches or overlaps zone A. Continuing with thisexample, the display characteristic indicator 925C indicates that therespective motion event corresponding to the representation 998A touchesor overlaps zone A.

In FIG. 9Y, the motion events feed 997 also includes: an exit affordance999, which, when activated (e.g., via a tap gesture), causes the clientdevice 504 to display a previous user interface (e.g., the videomonitoring UI in FIG. 9X); and a filtering affordance 9100, which, whenactivated (e.g., via a tap gesture), causes the client device 504 todisplay a filtering pane (e.g., the filtering pane 9105 in FIG. 9AA). InFIG. 9Y, the motion events feed 997 further includes a scroll bar 9101for viewing the balance of the representations 998 in the motion eventsfeed 997. FIG. 9Y also illustrates client device 504 detecting an upwarddragging gesture on the touch screen 906 whereby a contact 9102 is movedfrom a first location 9103A to a second location 9103B.

FIG. 9Z illustrates the client device 504 displaying a second portion ofthe motion events feed 997 in response to detecting the upward dragginggesture in FIG. 9Y. The second portion of the motion events feed 997 inFIG. 9Z shows a second set of representations 998 that are distinct fromthe first set of representations 998 shown in the first portion of themotion events feed 997 in FIG. 9Y. FIG. 9Z also illustrates the clientdevice 504 detecting a contact 9104 at a location corresponding to thefiltering affordance 9100 on the touch screen 906.

FIG. 9AA illustrates the client device 504 displaying a filtering pane9105 in response to detecting selection of the filtering affordance 9100in FIG. 9Z. In FIG. 9AA, the filtering pane 9105 includes a list ofcategories with recognized event categories and previously created zonesof interest. The filtering pane 9105 includes an entry 924A forrecognized event category A, including: a display characteristicindicator 925A representing the display characteristic forrepresentations corresponding to motion events assigned to eventcategory A (e.g., vertical stripes), an indicator filter 926A forenabling/disabling display of representations 998 in the motion eventsfeed 997 for motion events assigned to event category A; a notificationsindicator 927A for enabling/disabling notifications sent in response todetection of motion events assigned to event category A; and an “EditCategory” affordance 9106A for displaying an editing user interface (UI)for event category A. The filtering pane 9105 also includes an entry924B for recognized event category “Birds in Flight,” including: adisplay characteristic indicator 925B representing the displaycharacteristic for representations corresponding to motion eventsassigned to “Birds in Flight” (e.g., a diagonal shading pattern); anindicator filter 926B for enabling/disabling display of representations998 in the motion events feed 997 for motion events assigned to “Birdsin Flight”; a notifications indicator 927B for enabling/disablingnotifications sent in response to detection of motion events assigned to“Birds in Flight”; and an “Edit Category” affordance 9106B fordisplaying an editing UI for “Birds in Flight.”

In FIG. 9AA, the filtering pane 9105 further includes an entry 924C forzone A, including: a display characteristic indicator 925C representingthe display characteristic for representations corresponding to motionevents that touch or overlap zone A (e.g., an ‘X’ at the bottom of theevent indicator); an indicator filter 926C for enabling/disablingdisplay of representations 998 in the motion events feed 997 for motionevents that touch or overlap zone A; a notifications indicator 927C forenabling/disabling notifications sent in response to detection of motionevents that touch or overlap zone A; and an “Edit Category” affordance9106C for displaying an editing UI for the zone A category. Thefiltering pane 9105 further includes an entry 924D for uncategorizedmotion events, including: a display characteristic indicator 925Drepresenting the display characteristic for representationscorresponding to uncategorized motion events (e.g., an event indicatorwithout fill or shading); an indicator filter 926D forenabling/disabling display of representations 998 in the motion eventsfeed 997 for uncategorized motion events assigned; a notificationsindicator 927D for enabling/disabling notifications sent in response todetection of uncategorized motion events; and an “Edit Category”affordance 9106D for displaying an editing UI for the unrecognizedcategory. FIG. 9AA also illustrates client device 504 detecting acontact 9107 at a location corresponding to the “Edit Category”affordance 9106C on the touch screen 906.

FIG. 9BB illustrates the client device 504 displaying an editing UI forthe zone A category in response to detecting selection of the “EditCategory” affordance 9106C in FIG. 9AA. In FIG. 9BB, the editing UI forthe zone A category includes two distinct regions: a first region 9112;and a second region 9114. The first region 9114 includes: a label textentry box 9114 for renaming the label for the zone A category from thedefault name (“zone A”) to a custom name; and an “Edit Indicator DisplayCharacteristic” affordance 9116 for editing the default displaycharacteristic 925C for representations corresponding to motion eventsthat touch or overlap zone A (e.g., from the ‘X’ at the bottom of theevent indicator to a fill color or shading pattern). The first region9114 also includes: a notifications indicator 927C forenabling/disabling notifications sent in response to detection of motionevents that touch or overlap zone A; and a save/exit affordance 9118 forsaving changes made to the zone A category or exiting the editing UI forthe zone A category.

In FIG. 9BB, the second region 9112 includes representations 998(sometimes also herein called “sprites”) of motion events that touch oroverlap zone A, where a respective representation 998A corresponds to amotion event that touches or overlaps zone A. In some implementations,the respective representation 998A includes a series of frames or avideo clips of the motion event that touches or overlaps zone A. Forexample, in FIG. 9BB, the respective representation 998A corresponds toa motion event of a jackrabbit running from right to left across thefield of view of the respective camera at least partially within zone A.In FIG. 9BB, the respective representation 998A is associated with acheckbox 9120. In some implementations, when the checkbox 9120 isunchecked (e.g., with a tap gesture) the motion event corresponding tothe checkbox 9120 is removed the zone A category.

Client-Side Zooming of a Remote Video Feed

FIG. 10 is a flow diagram of a process 1000 for performing client-sidezooming of a remote video feed in accordance with some implementations.In some implementations, the process 1000 is performed at least in partby a server with one or more processors and memory, a client device withone or more processors and memory, and a camera with one or moreprocessors and memory. For example, in some implementations, the serveris the video server system 508 (FIGS. 5-6) or a component thereof (e.g.,server-side module 506, FIGS. 5-6), the client device is the clientdevice 504 (FIGS. 5 and 7) or a component thereof (e.g., the client-sidemodule 502, FIGS. 5 and 7), and the camera is a respective one of one ormore camera 118 (FIGS. 5 and 8).

In some implementations, control and access to the smart homeenvironment 100 is implemented in the operating environment 500 (FIG. 5)with a video server system 508 (FIGS. 5-6) and a client-side module 502(FIGS. 5 and 7) (e.g., an application for monitoring and controlling thesmart home environment 100) is executed on one or more client devices504 (FIGS. 5 and 7). In some implementations, the video server system508 manages, operates, and controls access to the smart home environment100. In some implementations, a respective client-side module 502 isassociated with a user account registered with the video server system508 that corresponds to a user of the client device 504.

The server maintains (1002) the current digital tilt-pan-zoom (DTPZ)settings for the camera. In some implementations, the server storesvideo settings (e.g., tilt, pan, and zoom settings) for each of the oneor more cameras 118 associated with the smart home environment 100.

The camera sends (1004) a video feed at the current DTPZ settings to theserver. The server sends (1006) the video feed to the client device. Insome implementations, the camera directly sends the video feed to theclient device.

The client device presents (1008) the video feed on an associateddisplay. FIG. 9A, for example, shows the client device 504 displaying afirst implementation of the video monitoring user interface (UI) of theapplication on the touch screen 906. In FIG. 9A, the video monitoring UIincludes three distinct regions: a first region 903, a second region905, and a third region 907. In FIG. 9A, the first region 903 includes avideo feed from a respective camera among the one or more camera 118associated with the smart home environment 100. For example, therespective camera is located on the back porch of the user's domicile orpointed out of a window of the user's domicile. In FIG. 9A, for example,an indicator 912 indicates that the video feed being displayed in thefirst region 903 is a live video feed.

The client device detects (1010) a first user input. FIG. 9R, forexample, shows the client device 504 detecting a pinch-in gesture withcontacts 965A and 965B (i.e., the first user input) relative to arespective portion of the video feed in the first region 903 of thevideo monitoring UI on the touch screen 906.

In response to detecting the first user input, the client deviceperforms (1012) a local software-based zoom on a portion of the videofeed according to the first user input. FIG. 9S, for example, shows theclient device 504 displaying a zoomed-in portion of the video feed inresponse to detecting the pinch-in gesture (i.e., the first user input)on the touch screen 906 in FIG. 9R. In some implementations, thezoomed-in portion of the video feed corresponds to a software-based zoomperformed locally by the client device 504 on the respective portion ofthe video feed corresponding to the pinch-in gesture in FIG. 9R.

The client device detects (1014) a second user input. In FIG. 9S, forexample, the video controls in the first region 903 further includes anenhancement affordance 968 in response to detecting the pinch-in gesture(i.e., the first user input) in FIG. 9R. FIG. 9S, for example, shows theclient device 504 detecting a contact 967 (i.e., the second user input)at a location corresponding to the enhancement affordance 968 on thetouch screen 906.

In response to detecting the second user input, the client devicedetermines (1016) the current zoom magnification and coordinates of thezoomed-in portion of the video feed. In some implementations, the clientdevice 504 or a component thereof (e.g., camera control module 732, FIG.7) determines the zoom magnification of the local, software zoomfunction and the coordinates of the respective portion of the video feedin response to detecting the contact 967 (i.e., the second user input)in FIG. 9S.

The client device sends (1018) a zoom command to the server includingthe current zoom magnification and the coordinates. In someimplementations, the client device 504 or a component thereof (e.g.,camera control module 732, FIG. 7) causes the command to be sent to therespective camera, where the command includes the current zoommagnification of the software zoom function and coordinates of therespective portion of the first video feed. In some implementations, thecommand is typically relayed through the video server system 508 or acomponent thereof (e.g., the camera control module 618, FIG. 6) to therespective camera. In some implementations, however, the client device504 sends the command directly to the respective camera.

In response to receiving the zoom command, the server changes (1020) thestored DTPZ settings for the camera based on the zoom command. In someimplementations, the server changes the stored video settings (e.g.,tilt, pan, and zoom settings) for the respective camera according to thezoom command. In response to receiving the zoom command, the serversends (1022) the zoom command to the camera including the zoommagnification and the coordinates.

In response to receiving the zoom command, the camera performs (1024) ahardware-based zoom according to the zoom magnification and thecoordinates. The respective camera performs a hardware zoom at the zoommagnification on the coordinates indicated by the zoom command. Thus,the respective camera crops its field of view to the coordinatesindicated by the zoom command.

After performing the hardware-based zoom, the camera sends (1026) thechanged video feed to the server. The respective camera sends thechanged video feed with the field of view corresponding to thecoordinates indicated by the zoom command. The server sends (1028) thechanged video feed to the client device. In some implementations, thecamera directly sends the changed video feed to the client device.

The client device presents (1030) the changed video feed on theassociated display. FIG. 9U, for example, shows the client device 504displaying the changed video feed at a higher resolution as compared toFIG. 9S, where the local, software zoom produced a lower resolution ofthe respective portion.

It should be understood that the particular order in which theoperations in FIG. 10 have been described is merely an example and isnot intended to indicate that the described order is the only order inwhich the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods and/or processesdescribed herein (e.g., the methods 1200, 1300, 1400, 1500, and 1600)are also applicable in an analogous manner to the method 1000 describedabove with respect to FIG. 10.

System Architecture and Data Processing Pipeline

FIG. 11A illustrates a representative system architecture 1102 and acorresponding data processing pipeline 1104. The data processingpipeline 1104 processes a live video feed received from a video source522 (e.g., including a camera 118 and an optional controller device) inreal-time to identify and categorize motion events in the live videofeed, and sends real-time event alerts and a refreshed event timeline toa client device 504 associated with a reviewer account bound to thevideo source 522.

In some implementations, after video data is captured at the videosource 522, the video data is processed to determine if any potentialmotion event candidates are present in the video stream. A potentialmotion event candidate detected in the video data is also referred to asa cue point. Thus, the initial detection of motion event candidates isalso referred to as cue point detection. A detected cue point triggersperformance of a more through event identification process on a videosegment corresponding to the cue point. In some implementations, themore through event identification process includes obtaining the videosegment corresponding to the detected cue point, background estimationfor the video segment, motion object identification in the videosegment, obtaining motion tracks for the identified motion object(s),and motion vector generation based on the obtained motion tracks. Theevent identification process may be performed by the video source 522and the video server system 508 cooperatively, and the division of thetasks may vary in different implementations, for different equipmentcapability configurations, and/or for different network and server loadsituations. After the motion vector for the motion event candidate isobtained, the video server system 508 categorizes the motion eventcandidate, and presents the result of the event detection andcategorization to a reviewer associated with the video source 522.

In some implementations, the video server system 508 includes functionalmodules for an event preparer, an event categorizer, and a user facingfrontend. The event preparer obtains the motion vectors for motion eventcandidates (e.g., by processing the video segment corresponding to a cuepoint or by receiving the motion vector from the video source). Theevent categorizer categorizes the motion event candidates into differentevent categories. The user facing frontend generates event alerts andfacilitates review of the motion events by a reviewer through a reviewinterface on a client device 504. The client facing frontend alsoreceives user edits on the event categories, user preferences for alertsand event filters, and zone definitions for zones of interest. The eventcategorizer optionally revises event categorization models and resultsbased on the user edits received by the user facing frontend.

In some implementations, the video server system 508 also determines anevent mask for each motion event candidate and caches the event mask forlater use in event retrieval based on selected zone(s) of interest.

In some implementations, the video server system 508 stores raw orcompressed video data (e.g., in a video data database 1106), eventcategorization model (e.g., in an event categorization model database1108), and event masks and other event metadata (e.g., in an event dataand event mask database 1110) for each of the video sources 522.

The above is an overview of the system architecture 1102 and the dataprocessing pipeline 1104 for event processing in video monitoring. Moredetails of the processing pipeline and processing techniques areprovided below.

As shown in the upper portion of FIG. 11A, the system architecture 1102includes the video source 522. The video source 522 transmits a livevideo feed to the remote video server system 508 via one or morenetworks (e.g., the network(s) 162). In some implementations, thetransmission of the video data is continuous as the video data iscaptured by the camera 118. In some implementations, the transmission ofvideo data is irrespective of the content of the video data, and thevideo data is uploaded from the video source 522 to the video serversystem 508 for storage irrespective of whether any motion event has beencaptured in the video data. In some implementations, the video data maybe stored at a local storage device of the video source 522 by default,and only video segments corresponding to motion event candidatesdetected in the video stream are uploaded to the video server system 508in real-time.

In some implementations, the video source 522 dynamically determineswhich parts of the video stream are to be uploaded to the video serversystem 508 in real-time. For example, in some implementations, dependingon the current server load and network conditions, the video source 522optionally prioritizes the uploading of video segments correspondingnewly detected motion event candidates ahead of other portions of thevideo stream that do not contain any motion event candidates. Thisupload prioritization helps to ensure that important motion events aredetected and alerted to the reviewer in real-time, even when the networkconditions and server load are less than optimal. In someimplementations, the video source 522 implements two parallel uploadconnections, one for uploading the continuous video stream captured bythe camera 118, and the other for uploading video segments correspondingdetected motion event candidates. At any given time, the video source522 determines whether the uploading of the continuous video streamneeds to be suspended temporarily to ensure that sufficient bandwidth isgiven to the uploading of the video segments corresponding to newlydetected motion event candidates.

In some implementations, the video stream uploaded for cloud storage isat a lower quality (e.g., lower resolution, lower frame rate, highercompression, etc.) than the video segments uploaded for motion eventprocessing.

As shown in FIG. 11A, the video source 522 includes a camera 118, and anoptional controller device. In some implementations, the camera 118includes sufficient on-board processing power to perform all necessarylocal video processing tasks (e.g., cue point detection for motion eventcandidates, video uploading prioritization, network connectionmanagement, etc.), and the camera 118 communicates with the video serversystem 508 directly, without any controller device acting as anintermediary. In some implementations, the camera 118 captures the videodata and sends the video data to the controller device for the necessarylocal video processing tasks. The controller device optionally performsthe local processing tasks for more than one camera 118. For example,there may be multiple cameras in one smart home environment (e.g., thesmart home environment 100, FIG. 1), and a single controller devicereceives the video data from each camera and processes the video data todetect motion event candidates in the video stream from each camera. Thecontroller device is responsible for allocating sufficient outgoingnetwork bandwidth to transmitting video segments containing motion eventcandidates from each camera to the server before using the remainingbandwidth to transmit the video stream from each camera to the videoserver system 508. In some implementations, the continuous video streamis sent and stored at one server facility while the video segmentscontaining motion event candidates are send to and processed at adifferent server facility.

As shown in FIG. 11A, after video data is captured by the camera 118,the video data is optionally processed locally at the video source 522in real-time to determine whether there are any cue points in the videodata that warrant performance of a more thorough event identificationprocess. Cue point detection is a first layer motion eventidentification which is intended to be slightly over-inclusive, suchthat real motion events are a subset of all identified cue points. Insome implementations, cue point detection is based on the number ofmotion pixels in each frame of the video stream. In someimplementations, any method of identifying motion pixels in a frame maybe used. For example, a Gaussian mixture model is optionally used todetermine the number of motion pixels in each frame of the video stream.In some implementations, when the total number of motion pixels in acurrent image frame exceeds a predetermined threshold, a cue point isdetected. In some implementations, a running sum of total motion pixelcount is calculated for a predetermined number of consecutive frames aseach new frame is processed, and a cue point is detected when therunning sum exceeds a predetermined threshold. In some implementations,as shown in FIG. 11B-(a), a profile of total motion pixel count overtime is obtained. In some implementations, a cue point is detected whenthe profile of total motion pixel count for a current frame sequence ofa predetermined length (e.g., 30 seconds) meets a predetermined triggercriterion (e.g., total pixel count under the profile>a threshold motionpixel count).

In some implementations, the beginning of a cue point is the time whenthe total motion pixel count meets a predetermined threshold (e.g., 50motion pixels). In some implementations, the start of the motion eventcandidate corresponding to a cue point is the beginning of the cue point(e.g., t1 in FIG. 11B-(a)). In some implementations, the start of themotion event candidate is a predetermined lead time (e.g., 5 seconds)before the beginning of the cue point. In some implementations, thestart of a motion event candidate is used to retrieve a video segmentcorresponding to the motion event candidate for a more thorough eventidentification process.

In some implementations, the thresholds for detecting cue points areadjusted overtime based on performance feedback. For example, if toomany false positives are detected, the threshold for motion pixel countis optionally increased. If too many motion events are missed, thethreshold for motion pixel count is optionally decreased.

In some implementations, before the profile of the total motion pixelcount for a frame sequence is evaluated for cue point detection, theprofile is smoothed to remove short dips in total motion pixel count, asshown in FIG. 11B-(b). In general, once motion has started, momentarystops or slowing downs may occur during the motion, and such momentarystops or slowing downs are reflected as short dips in the profile oftotal motion pixel count. Removing these short dips from the profilehelps to provide a more accurate measure of the extent of motion for cuepoint detection. Since cue point detection is intended to be slightlyover-inclusive, by smoothing out the motion pixel profile, cue pointsfor motion events that contain momentary stops or slowing downs of themoving objects would less likely be missed by the cue point detection.

In some implementations, a change in camera state (e.g., IR mode, AEmode, DTPZ settings, etc.) may changes pixel values in the image framesdrastically even though no motion has occurred in the scene captured inthe video stream. In some implementations, each camera state change isnoted in the cue point detection process (as shown in FIG. 11B-(c)), anda detected cue point is optionally suppressed if its occurrence overlapswith one of the predetermined camera state changes. In someimplementations, the total motion pixel count in each frame is weigheddifferently if accompanied with a camera state change. For example, thetotal motion pixel count is optionally adjusted by a fraction (e.g.,10%) if it is accompanied by a camera state change, such as an IR modeswitch. In some implementations, the motion pixel profile is reset aftereach camera state change.

Sometimes, a fast initial increase in total motion pixel count mayindicate a global scene change or a lighting change, e.g., when thecurtain is drawn, or when the camera is pointed in a different directionor moved to a different location by a user. In some implementations, asshown in FIG. 11B-(d), when the initial increase in total motion pixelcount in the profile of total motion pixel count exceeds a predeterminedrate, a detected cue point is optionally suppressed. In someimplementations, the suppressed cue point undergoes an edge caserecovery process to determine whether the cue point is in fact not dueto lighting change or camera movement, but rather a valid motion eventcandidate that needs to be recovered and reported for subsequent eventprocessing. In some implementations, the profile of motion pixel countis reset when such fast initial increase in total motion pixel count isdetected and a corresponding cue point is suppressed.

In some implementations, the cue point detection generally occurs at thevideo source 522, and immediately after a cue point is detected in thelive video stream, the video source 522 sends an event alert to thevideo server system 508 to trigger the subsequent event processing. Insome implementations, the video source 522 includes a video camera withvery limited on-board processing power and no controller device, and thecue point detection described herein is performed by the video serversystem 508 on the continuous video stream transmitted from the camera tothe video server system 508.

In some implementations, after a cue point is detected in the videostream, a video segment corresponding to the cue point is used toidentify a motion track of a motion object in the video segment. Theidentification of motion track is optionally performed locally at thevideo source 522 or remotely at the video server system 508. In someimplementations, the identification of the motion track based on a videosegment corresponding to a detected cue point is performed at the videoserver system 508 by an event preparer module. In some implementations,the event preparer module receives an alert for a cue point detected inthe video stream, and retrieves the video segment corresponding to thecue point from cloud storage (e.g., the video data database 1106, FIG.11A) or from the video source 522. In some implementations, the videosegment used to identify the motion track may be of higher quality thanthe video uploaded for cloud storage, and the video segment is retrievedfrom the video source 522 separately from the continuous video feeduploaded from the video source 522.

In some implementations, after the event preparer module obtains thevideo segment corresponding to a cue point, the event preparer moduleperforms background estimation, motion object identification, and motiontrack determination. Once the motion track(s) of the motion object(s)identified in the video segment are determined, the event preparermodule generates a motion vector for each of the motion object detectedin the video segment. Each motion vector corresponds to one motion eventcandidate. In some implementations, false positive suppression isoptionally performed to reject some motion event candidates before themotion event candidates are submitted for event categorization.

In some implementations, if the video source 522 has sufficientprocessing capabilities, the background estimation, motion trackdetermination, and the motion vector generation are optionally performedlocally at the video source 522.

In some implementations, the motion vector representing a motion eventcandidate is a simple two-dimensional linear vector defined by a startcoordinate and an end coordinate of a motion object in a scene depictedin the video segment, and the motion event categorization is based onthe simple two-dimensional linear motion vector. The advantage of usingthe simple two-dimensional linear motion vector for event categorizationis that the event data is very compact, and fast to compute and transmitover a network. When network bandwidth and/or server load isconstrained, simplifying the representative motion vector andoff-loading the motion vector generation from the event preparer moduleof the video server system 508 to the video source 522 can help torealize the real-time event categorization and alert generation for manyvideo sources in parallel.

In some implementations, after motion tracks in a video segmentcorresponding to a cue point are determined, track lengths for themotion tracks are determined. In some implementations, “short tracks”with track lengths smaller than a predetermined threshold (e.g., 8frames) are suppressed, as they are likely due to trivial movements,such as leaves shifting in the wind, water shimmering in the pond, etc.In some implementations, pairs of short tracks that are roughly oppositein direction are suppressed as “noisy tracks.” In some implementations,after the track suppression, if there are no motion tracks remaining forthe video segment, the cue point is determined to be a false positive,and no motion event candidate is sent to the event categorizer for eventcategorization. If at least one motion track remains after the falsepositive suppression is performed, a motion vector is generated for eachremaining motion track, and corresponds to a respective motion eventcandidate going into event categorization. In other words, multiplemotion event candidates may be generated based on a video segment, whereeach motion event candidate represents the motion of a respective motionobject detected in the video segment. The false positive suppressionoccurring after the cue point detection and before the motion vectorgeneration is the second layer false positive suppression, which removesfalse positives based on the characteristics of the motion tracks.

In some implementations, object identification is performed bysubtracting the estimated background from each frame of the videosegment. A foreground motion mask is then obtained by masking all pixellocations that have no motion pixels. An example of a motion mask isshown in FIG. 11C-(a). The example motion mask shows the motion pixelsin one frame of the video segment in white, and the rest of the pixelsin black. Once motion objects are identified in each frame, the samemotion object across multiple frames of the video segment are correlatedthrough a matching algorithm (e.g., Hungarian matching algorithm), and amotion track for the motion object is determined based on the “movement”of the motion object across the multiple frames of the video segment.

In some implementations, the motion track is used to generate atwo-dimensional linear motion vector which only takes into account thebeginning and end locations of the motion track (e.g., as shown by thedotted arrow in FIG. 11C-(b)). In some implementations, the motionvector is a non-linear motion vector that traces the entire motion trackfrom the first frame to the last frame of the frame sequence in whichthe motion object has moved.

In some implementations, the motion masks corresponding to each motionobject detected in the video segment are aggregated across all frames ofthe video segment to create an event mask for the motion event involvingthe motion object. As shown in FIG. 11C-(b), in the event mask, allpixel locations containing less than a threshold number of motion pixels(e.g., one motion pixel) are masked and shown in black, while all pixellocations containing at least the threshold number of motion pixels areshown in white. The active portion of the event mask (e.g., shown inwhite) indicates all areas in the scene depicted in the video segmentthat have been accessed by the motion object during its movement in thescene. In some implementations, the event mask for each motion event isstored at the video server system 508 or a component thereof (e.g., thezone creation module 624, FIG. 6), and used to selectively retrievemotion events that enter or touch a particular zone of interest withinthe scene depicted in the video stream of a camera. More details on theuse of event masks are provided later in the present disclosure withrespect to real-time zone monitoring, and retroactive eventidentification for newly created zones of interest.

In some implementations, a motion mask is created based on anaggregation of motion pixels from a short frame sequence in the videosegment. The pixel count at each pixel location in the motion mask isthe sum of the motion pixel count at that pixel location from all framesin the short frame sequence. All pixel locations in the motion mask withless than a threshold number of motion pixels (e.g., motion pixelcount >4 for 10 consecutive frames) are masked. Thus, the unmaskedportions of the motion mask for each such short frame sequence indicatesa dominant motion region for the short frame sequence. In someimplementations, a motion track is optionally created based on the pathtaken by the dominant motion regions identified from a series ofconsecutive short frame sequences.

In some implementations, an event mask is optionally generated byaggregating all motion pixels from all frames of the video segment ateach pixel location, and masking all pixel locations that have less thana threshold number of motion pixels. The event mask generated this wayis no longer a binary event mask, but is a two-dimensional histogram.The height of the histogram at each pixel location is the sum of thenumber of frames that contain a motion pixel at that pixel location.This type of non-binary event mask is also referred to as a motionenergy map, and illustrates the regions of the video scene that are mostactive during a motion event. The characteristics of the motion energymaps for different types of motion events are optionally used todifferentiate them from one another. Thus, in some implementations, themotion energy map of a motion event candidate is vectorized to generatethe representative motion vector for use in event categorization. Insome implementations, the motion energy map of a motion event isgenerated and cached by the video server system and used for real-timezone monitoring, and retro-active event identification for newly createdzones of interest.

In some implementations, a live event mask is generated based on themotion masks of frames that have been processed, and is continuouslyupdated until all frames of the motion event have been processed. Insome implementations, the live event mask of a motion event in progressis used to determine if the motion event is an event of interest for aparticular zone of interest. More details of how a live event mask isused for zone monitoring are provided later in the present disclosure.

In some implementations, after the video server system 508 obtains therepresentative motion vector for a new motion event candidate (e.g.,either by generating the motion vector from the video segmentcorresponding to a newly detected cue point), or by receiving the motionvector from the video source 522, the video server system 508 proceedsto categorize the motion event candidate based on its representativemotion vector.

Motion Event Categorization and Retroactive Activity Recognition

In some implementations, the categorization of motion events (alsoreferred to as “activity recognition”) is performed by training acategorization model based on a training data set containing motionvectors corresponding to various known event categories (e.g., personrunning, person jumping, person walking, dog running, car passing by,door opening, door closing, etc.). The common characteristics of eachknown event category that distinguish the motion events of the eventcategory from motion events of other event categories are extractedthrough the training. Thus, when a new motion vector corresponding to anunknown event category is received, the event categorizer moduleexamines the new motion vector in light of the common characteristics ofeach known event category (e.g., based on a Euclidean distance betweenthe new motion vector and a canonical vector representing each knownevent type), and determines the most likely event category for the newmotion vector among the known event categories.

Although motion event categorization based on pre-established motionevent categories is an acceptable way to categorize motion events, thiscategorization technique may only be suitable for use when the varietyof motion events handled by the video server system 508 is relativelyfew in number and already known before any motion event is processed. Insome implementations, the video server system 508 serves a large numberof clients with cameras used in many different environmental settings,resulting in motion events of many different types. In addition, eachreviewer may be interested in different types of motion events, and maynot know what types of events they would be interested in before certainreal world events have happened (e.g., some object has gone missing in amonitored location). Thus, it is desirable to have an eventcategorization technique that can handle any number of event categoriesbased on actual camera use, and automatically adjust (e.g., create andretire) event categories through machine learning based on the actualvideo data that is received over time.

In some implementations, categorization of motion events is through adensity-based clustering technique (e.g., DBscan) that forms clustersbased on density distributions of motion events (e.g., motion events asrepresented by their respective motion vectors) in a vector event space.Regions with sufficiently high densities of motion vectors are promotedas recognized event categories, and all motion vectors within eachpromoted region are deemed to belong to a respective recognized eventcategory associated with that promoted region. In contrast, regions thatare not sufficiently dense are not promoted or recognized as eventcategories. Instead, such non-promoted regions are collectivelyassociated with a category for unrecognized events, and all motionvectors within such non-promoted regions are deemed to be unrecognizedmotion events at the present time.

In some implementations, each time a new motion vector comes in to becategorized, the event categorizer places the new motion vector into thevector event space according to its value. If the new motion vector issufficiently close to or falls within an existing dense cluster, theevent category associated with the dense cluster is assigned to the newmotion vector. If the new motion vector is not sufficiently close to anyexisting cluster, the new motion vector forms its own cluster of onemember, and is assigned to the category of unrecognized events. If thenew motion vector is sufficiently close to or falls within an existingsparse cluster, the cluster is updated with the addition of the newmotion vector. If the updated cluster is now a dense cluster, theupdated cluster is promoted, and all motion vectors (including the newmotion vector) in the updated cluster are assigned to a new eventcategory created for the updated cluster. If the updated cluster isstill not sufficiently dense, no new category is created, and the newmotion vector is assigned to the category of unrecognized events. Insome implementations, clusters that have not been updated for at least athreshold expiration period are retired. The retirement of old staticclusters helps to remove residual effects of motion events that are nolonger valid, for example, due to relocation of the camera that resultedin a scene change.

FIG. 11D illustrates an example process for the event categorizer of thevideo server system 508 to (1) gradually learn new event categoriesbased on received motion events, (2) assign newly received motion eventsto recognized event categories or an unrecognized event category, and(3) gradually adapt the recognized event categories to the more recentmotion events by retiring old static clusters and associated eventcategories, if any. The example process is provided in the context of adensity-based clustering algorithm (e.g., sequential DBscan). However, aperson skilled in the art will recognize that other clusteringalgorithms that allow growth of clusters based on new vector inputs canalso be used in various implementations.

As a background, sequential DB scan allows growth of a cluster based ondensity reachability and density connectedness. A point q is directlydensity-reachable from a point p if it is not farther away than a givendistance ϵ (i.e., is part of its ϵ-neighborhood) and if p is surroundedby sufficiently many points M such that one may consider p and q to bepart of a cluster. q is called density-reachable from p if there is asequence p₁, . . . p_(n) of points with p₁=p and p_(n)=p where eachp_(i+1) is directly density-reachable from p_(i). Since the relation ofdensity-reachable is not symmetric, another notion ofdensity-connectedness is introduced. Two points p and q aredensity-connected if there is a point o such that both p and q aredensity-reachable from o. Density-connectedness is symmetric. A clusteris defined by two properties: (1) all points within the cluster aremutually density-connected, and (2) if a point is density-reachable fromany point of the cluster, it is part of the cluster as well. Theclusters formed based on density connectedness and density reachabilitycan have all shapes and sizes, in other words, motion event candidatesfrom a video source (e.g., as represented by motion vectors in adataset) can fall into non-linearly separable clusters based on thisdensity-based clustering algorithm, when they cannot be adequatelyclustered by K-means or Gaussian Mixture EM clustering techniques. Insome implementations, the values of c and M are adjusted by the videoserver system 508 for each video source or video stream, such thatclustering quality can be improved for different camera usage settings.

In some implementations, during the categorization process, fourparameters are stored and sequentially updated for each cluster. Thefour parameters include: (1) cluster creation time, (2) cluster weight,(3) cluster center, and (4) cluster radius. The creation time for agiven cluster records the time when the given cluster was created. Thecluster weight for a given cluster records a member count for thecluster. In some implementations, a decay rate is associated with themember count parameter, such that the cluster weight decays over time ifan insufficient number of new members are added to the cluster duringthat time. This decaying cluster weight parameter helps to automaticallyfade out old static clusters that are no longer valid. The clustercenter of a given cluster is the weighted average of points in the givencluster. The cluster radius of a given cluster is the weighted spread ofpoints in the given cluster (analogous to a weighted variance of thecluster). It is defined that clusters have a maximum radius of ϵ/2. Acluster is considered to be a dense cluster when it contains at leastM/2 points. When a new motion vector comes into the event space, if thenew motion vector is density-reachable from any existing member of agiven cluster, the new motion vector is included in the existingcluster; and if the new motion vector is not density-reachable from anyexisting member of any existing cluster in the event space, the newmotion vector forms its own cluster. Thus, at least one cluster isupdated or created when a new motion vector comes into the event space.

FIG. 11D-(a) shows the early state of the event vector space 1114. Attime ti, two motion vectors (e.g., represented as two points) have beenreceived by the event categorizer. Each motion vector forms its owncluster (e.g., c₁ and c₂, respectively) in the event space 1114. Therespective creation time, cluster weight, cluster center, and clusterradius for each of the two clusters are recorded. At this time, norecognized event category exists in the event space, and the motionevents represented by the two motion vectors are assigned to thecategory of unrecognized events. On the frontend, the event indicatorsof the two events indicate that they are unrecognized events on theevent timeline, for example, in the manner shown in FIG. 9C.

After some time, a new motion vector is received and placed in the eventspace 1114 at time t₂. As shown in FIG. 11D-(b), the new motion vectoris density-reachable from the existing point in cluster c₂ and thusfalls within the existing cluster c₂. The cluster center, clusterweight, and cluster radius of cluster c₂ are updated based on the entryof the new motion vector. The new motion vector is also assigned to thecategory of unrecognized events. In some implementations, the eventindicator of the new motion event is added to the event timeline inreal-time, and has the appearance associated with the category forunrecognized events.

FIG. 11D-(c) illustrates that, at time t₃, two new clusters c₃ and c₄have been established and grown in size (e.g., cluster weight andradius) based on a number of new motion vectors received during the timeinterval between t₂ and t₃. In the meantime, neither cluster c₁ norcluster c₂ have seen any growth. The cluster weights for clusters c₁ andc₂ have decayed gradually due to the lack of new members during thisperiod of time. Up to this point, no recognized event category has beenestablished, and all motion events are assigned to the category ofunrecognized events. If the motion events are reviewed in a reviewinterface on the client device 504, the event indicators of the motionevents have an appearance associated with the category for unrecognizedevents (e.g., as the event indicators 922 show in FIGS. 9C). Each time anew motion event is added to the event space 1114, a corresponding eventindicator for the new event is added to the timeline associated with thepresent video source.

FIG. 11D-(d) illustrates that, at time t₄, another new motion vector hasbeen added to the event space 1114, and the new motion vector fallswithin the existing cluster C₃. The cluster center, cluster weight, andcluster radius of cluster c₃ are updated based on the addition of thenew motion vector, and the updated cluster c₃ has become a dense clusterbased on a predetermined density requirement (e.g., a cluster isconsidered dense when it contains at least M/2 points). Once cluster c₃has achieved the dense cluster status (and re-labeled as C₃), a newevent category is established for cluster C₃. When the new eventcategory is established for cluster C₃, all the motion vectors currentlywithin cluster C₃ are associated with the new event category. In otherwords, the previously unrecognized events in cluster C₃ are nowrecognized events of the new event category. In some implementations, assoon as the new event category is established, the event categorizernotifies the user facing frontend of the video server system 508 aboutthe new event category. The user facing frontend determines whether areviewer interface for the video stream corresponding to the event space1114 is currently displayed on a client device 504. If a reviewerinterface is currently displayed, the user facing frontend causes theclient device 504 to retroactively modify the display characteristics ofthe event indicators for the motion events in cluster C₃ to reflect thenewly established event category in the review interface. For example,as soon as the new event category is established by the eventcategorizer, the user facing frontend will cause the event indicatorsfor the motion events previously within cluster c₃ (and now in clusterC₃) to take on a color assigned to the new event category). In addition,the event indicator of the new motion event will also take on the colorassigned to the new event category. This is illustrated in the reviewinterface 908 in FIG. 9D by the changing color of the event indicators922A, 922C, 922D and 922E to reflect the newly established eventcategory (supposing that cluster C₃ corresponds to Event Cat. A here).

FIG. 11D-(e) illustrates that, at time t₅, two new motion vectors havebeen received in the interval between t₄ and t₅. One of the two newmotion vectors falls within the existing dense cluster C₃, and isassociated with the recognized event category of cluster C₃. Once themotion vector is assigned to cluster C₃, the event categorizer notifiesthe user facing frontend regarding the event categorization result.Consequently, the event indicator of the motion event represented by thenewly categorized motion vector is given the appearance associated withthe recognized event category of cluster C₃. Optionally, a pop-upnotification for the newly recognized motion event is presented over thetimeline associated with the event space. This real-time recognition ofa motion event for an existing event category is illustrated in FIG. 9E,where an event indicator 922L and pop-up notification 928 for a newmotion event are shown to be associated with an existing event category“Event Cat. B” (supposing that cluster C₃ corresponds to Event Cat. Bhere). It should be noted that, in FIG. 9E, the presentation of thepop-up 928 and the retroactive coloring of the event indicators forEvent Cat. B can also happen at the time that when Event Cat. B becomesa newly recognized category upon the arrival of the new motion event.

FIG. 11D-(e) further illustrates that, at time t₅, one of the two newmotion vectors is density reachable from both of the existing clustersc₁ and c₅, and thus qualifies as a member for both clusters. The arrivalof this new motion vector halts the gradual decay in cluster weight thatcluster c₁ that has sustained since time t₁. The arrival of the newmotion vector also causes the existing clusters c₁ and c₅ to becomedensity-connected, and as a result, to merge into a larger cluster c₅.The cluster center, cluster weight, cluster radius, and optionally thecreation time for cluster c₅ are updated accordingly. At this time,cluster c₂ remains unchanged, and its cluster weight decays further overtime.

FIG. 11D-(f) illustrates that, at time t₆, the weight of the existingcluster c₂ has reached below a threshold weight, and is thus deletedfrom the event space 1114 as a whole. The pruning of inactive sparseclusters allows the event space to remain fairly noise-free and keepsthe clusters easily separable. In some implementations, the motionevents represented by the motion vectors in the deleted sparse clusters(e.g., cluster c₂) are retroactively removed from the event timeline onthe review interface. In some implementations, the motion eventsrepresented by the motion vectors in the deleted sparse clusters (e.g.,cluster c₂) are kept in the timeline and given a new appearanceassociated with a category for trivial or uncommon events. In someimplementations, the motion events represented by the motion vectors inthe deleted sparse cluster (e.g., cluster c₂) are optionally gatheredand presented to the user or an administrator to determine whether theyshould be removed from the event space and the event timeline.

FIG. 11D-(f) further illustrates that, at time t₆, a new motion vectoris assigned to the existing cluster c₅, which causes the cluster weight,cluster radius, and cluster center of cluster c₅ to be updatedaccordingly. The updated cluster c₅ now reaches the threshold forqualifying as a dense cluster, and is thus promoted to a dense clusterstatus (and relabeled as cluster C₅). A new event category is createdfor cluster C₅. All motion vectors in cluster C₅ (which were previouslyin clusters c₁ and c₄) are removed from the category for unrecognizedmotion events, and assigned to the newly created event category forcluster C₅. The creation of the new category and the retroactiveappearance change for the event indicators of the motion events in thenew category are reflected in the reviewer interface, and optionallynotified to the reviewer.

FIG. 11D-(g) illustrates that, at time t₇, cluster C₅ continues to growwith some of the subsequently received motion vectors. A new cluster c₆has been created and has grown with some of the subsequently receivedmotion vectors. Cluster c₃ has not seen any growth since time t₅, andits cluster weight has gradually decayed overtime.

FIG. 11D-(h) shows that, at a later time t₅, dense cluster C₃ is retired(deleted from the event space 1114) when its cluster weight has fallenbelow a predetermine cluster retirement threshold. In someimplementations, motion events represented by the motion vectors withinthe retired cluster C₃ are removed from the event timeline for thecorresponding video source. In some implementations, the motion eventsrepresented by the motion vectors as well as the retired event categoryassociated with the retired cluster C₃ are stored as obsolete motionevents, apart from the other more current motion events. For example,the video data and motion event data for obsolete events are optionallycompressed and archived, and require a recall process to reload into thetimeline. In some implementations, when an event category is retired,the event categorizer notifies the user facing frontend to remove theevent indicators for the motion events in the retired event categoryfrom the timeline. In some implementations, when an event category isretired, the motion events in the retired category are assigned to acategory for retired events and their event indicators are retroactivelygiven the appearance associated with the category for retired events inthe timeline.

FIG. 11D-(h) further illustrates that, at time t₈, cluster c₆ has grownsubstantially, and has been promoted as a dense cluster (relabeled ascluster C₆) and given its own event category. Thus, on the event reviewinterface, a new event category is provided, and the appearance of theevent indicators for motion events in cluster C₆ is retroactivelychanged to reflect the newly recognized event category.

Based on the above process, as motion vectors are collected in the eventspace overtime, the most common event categories emerge graduallywithout manual intervention. In some implementations, the creation of anew category causes real-time changes in the review interface providedto a client device 504 associated with the video source. For example, insome implementations, as shown in FIGS. 9A-9E, motion events are firstrepresented as uncategorized motion events, and as each event categoryis created overtime, the characteristics of event indicators for pastmotion events in that event category are changed to reflect the newlyrecognized event category. Subsequent motion events falling within therecognized categories also have event indicators showing theirrespective event categories. The currently recognized event categoriesare optionally presented in the review interface for user selection asevent filters. The user may choose any subset of the currently knownevent categories (e.g., each recognized event categories and respectivecategories for trivial events, rare events, obsolete events, andunrecognized events) to selectively view or receive notifications formotion events within the subset of categories. This is illustrated inFIGS. 9E-9G, where the user has selectively turned off the eventindicators for Event Cat. A and turned on the event indicators for EventCat. B on the timeline 910 by selecting Event Cat. B (via affordance926B) and deselecting Event Cat. A (via affordance 926A) in the region907. The real-time event notification is also turned off for Event Cat.A, and turned on for Event Cat. B by selecting Event Cat. B (viaaffordance 927B) and deselecting Event Cat. A (via affordance 927A) inthe third region 907.

In some implementations, a user may review past motion events and theircategories on the event timeline. In some implementations, the user isallowed to edit the event category assignments, for example, by removingone or more past motion events from a known event category, as shown inFIGS. 9H-9J. When the user has edited the event category composition ofa particular event category by removing one or more past motion eventsfrom the event category, the user facing frontend notifies the eventcategorizer of the edits. In some implementations, the event categorizerremoves the motion vectors of the removed motion events from the clustercorresponding to the event category, and re-computes the clusterparameters (e.g., cluster weight, cluster center, and cluster radius).In some implementations, the removal of motion events from a recognizedcluster optionally causes other motion events that are similar to theremoved motion events to be removed from the recognized cluster as well.In some implementations, manual removal of one or more motion eventsfrom a recognized category may cause one or more motion events to beadded to event category due to the change in cluster center and clusterradius. In some implementations, the event category models are stored inthe event category models database 1108 (FIG. 11A), and is retrieved andupdated in accordance with the user edits.

In some implementations, one event category model is established for onecamera. In some implementations, a composite model based on the motionevents from multiple related cameras (e.g., cameras reported to serve asimilar purpose, or have a similar scene, etc.) is created and used tocategorize motion events detected in the video stream of each of themultiple related cameras. In such implementations, the timeline for onecamera may show event categories discovered based on motion events inthe video streams of its related cameras, even though no event for suchcategories have been seen in the camera's own video stream.

Non-Causal Zone Search and Context-Aware Zone Monitoring

In some implementations, event data and event masks of past motionevents are stored in the event data and event mask database 1110 (FIG.11A). In some implementations, the client device 504 receives user inputto select one or more filters to selectively review past motion events,and selectively receive event alerts for future motion events.

In some implementations, the client device 504 passes the user selectedfilter(s) to the user facing frontend, and the user facing frontendretrieves the events of interest based on the information in the eventdata and event mask database 1110. In some implementations, theselectable filters include one or more recognized event categories, andoptionally any of the categories for unrecognized motion events, rareevents, and/or obsolete events. When a recognized event category isselected as a filter, the user facing frontend retrieves all past motionevents associated with the selected event category, and present them tothe user (e.g., on the timeline, or in an ordered list shown in a reviewinterface). For example, as shown in FIG. 9F-9G, when the user selectsone of the two recognized event categories in the review interface, thepast motion events associated with the selected event category (e.g.,Event Cat. B) are shown on the timeline 910, while the past motionevents associated with the unselected event category (e.g., Event Cat.A) are removed from the timeline. In another example, as shown in 9H-9J,when the user selects to edit a particular event category (e.g., EventCat. B), the past motion events associated with the selected eventcategories (e.g., Event Cat. B) are presented in the first region 935 ofthe editing user interface, while motion events in the unselected eventcategories (e.g., Event Cat. A) are not shown.

In some implementations, in addition to event categories, other types ofevent filters can also be selected individually or combined withselected event categories. For example, in some implementations, theselectable filters also include a human filter, which can be one or morecharacteristics associated with events involving a human being. Forexample, the one or more characteristics that can be used as a humanfilter include a characteristic shape (e.g., aspect ratio, size, shape,and the like) of the motion object, audio comprising human speech,motion objects having human facial characteristics, etc. In someimplementations, the selectable filters also include a filter based onsimilarity. For example, the user can select one or more example motionevents, and be presented one or more other past motion events that aresimilar to the selected example motion events. In some implementations,the aspect of similarity is optionally specified by the user. Forexample, the user may select “color content,” “number of moving objectsin the scene,” “shape and/or size of motion object,” and/or “length ofmotion track,” etc, as the aspect(s) by which similarity between twomotion events are measured. In some implementations, the user may chooseto combine two or more filters and be shown the motion events thatsatisfy all of the filters combined. In some implementations, the usermay choose multiple filters that will act separately, and be shown themotion events that satisfy at least one of the selected filters.

In some implementations, the user may be interested in past motionevents that have occurred within a zone of interest. The zone ofinterest can also be used as an event filter to retrieve past events andgenerate notifications for new events. In some implementations, the usermay define one or more zones of interest in a scene depicted in thevideo stream. For example, in the user interface shown in FIGS. 9L-9N,the user has defined a zone of interest 947 with any number of verticesand edges (e.g., four vertices and four edges) that is overlaid on thescene depicted in the video stream. The zone of interest may enclose anobject, for example, a chair, a door, a window, or a shelf, located inthe scene. Once a zone of interest is created, it is included as one ofthe selectable filters for selectively reviewing past motion events thathad entered or touched the zone. For example, as shown in FIG. 9N, oncethe user has created and selected the filter Zone A 924C, a past motionevent 922V which has touched Zone A is highlighted on the timeline 910,and includes an indicator (e.g., a cross mark) associated with thefilter Zone A. In addition, the user may also choose to receive alertsfor future events that enter Zone A, for example, by selecting the alertaffordance 927C associated with Zone A.

In some implementations, the video server system 508 (e.g., the userfacing frontend of the video server system 508) receives the definitionsof zones of interest from the client device 504, and stores the zones ofinterest in association with the reviewer account currently active onthe client device 504. When a zone of interest is selected as a filterfor reviewing motion events, the user facing frontend searches the eventdata database 1110 (FIG. 11A) to retrieve all past events that havemotion object(s) within the selected zone of interest. Thisretrospective search of event of interest can be performed irrespectiveof whether the zone of interest had existed before the occurrence of theretrieved past event(s). In other words, the user does not need to knowwhere in the scene he/she may be interested in monitoring before hand,and can retroactively query the event database to retrieve past motionevents based on a newly created zone of interest. There is norequirement for the scene to be divided into predefined zones first, andpast events be tagged with the zones in which they occur when the pastevents were first processed and stored.

In some implementations, the retrospective zone search based on newlycreated or selected zones of interest is implemented through a regulardatabase query where the relevant features of each past event (e.g.,which regions the motion object had entered during the motion event) aredetermined on the fly, and compared to the zones of interest. In someimplementations, the server optionally defines a few default zones ofinterest (e.g., eight (2×4) predefined rectangular sectors within thescene), and each past event is optionally tagged with the particulardefault zones of interest that the motion object has entered. In suchimplementations, the user can merely select one or more of the defaultzones of interest to retrieve the past events that touched or enteredthe selected default zones of interest.

In some implementations, event masks (e.g., the example event mask shownin FIG. 11C) each recording the extent of a motion region accessed by amotion object during a given motion event are stored in the event dataand event masks database 1110 (FIG. 11A). The event masks provide afaster and more efficient way of retrieving past motion events that havetouched or entered a newly created zone of interest.

In some implementations, the scene of the video stream is divided into agrid, and the event mask of each motion event is recorded as an array offlags that indicates whether motion had occurred within each gridlocation during the motion event. When the zone of interest includes atleast one of the grid location at which motion has occurred during themotion event, the motion event is deemed to be relevant to the zone ofinterest and is retrieved for presentation. In some implementations, theuser facing frontend imposes a minimum threshold on the number of gridlocations that have seen motion during the motion event, in order toretrieve motion events that have at least the minimum number of gridlocations that included motion. In other words, if the motion region ofa motion event barely touched the zone of interest, it may not beretrieved for failing to meet the minimum threshold on grid locationsthat have seen motion during the motion event.

In some implementations, an overlap factor is determined for the eventmask of each past motion event and a selected zone of interest, and ifthe overlapping factor exceeds a predetermined overlap threshold, themotion event is deemed to be a relevant motion event for the selectedzone of interest.

In some implementations, the overlap factor is a simple sum of alloverlapping grid locations or pixel locations. In some implementations,more weight is given to the central region of the zone of interest thanthe peripheral region of the zone of interest during calculation of theoverlap factor. In some implementations, the event mask is a motionenergy mask that stores the histogram of pixel count at each pixellocation within the event mask. In some implementations, the overlapfactor is weighted by the pixel count at the pixel locations that themotion energy map overlaps with the zone of interest.

By storing the event mask at the time that the motion event isprocessed, the retrospective search for motion events that are relevantto a newly created zone of interest can be performed relatively quickly,and makes the user experience for reviewing the events-of-interest moreseamless. As shown in FIG. 9N, creation of a new zone of interest, orselecting a zone of interest to retrieve past motion events that are notpreviously associated with the zone of interest provides many usagepossibilities, and greatly expands the utility of stored motion events.In other words, motion event data (e.g., event categories, event masks)can be stored in anticipation of different uses, without requiring suchuses to be tagged and stored at the time when the event occurs. Thus,wasteful storage of extra metadata tags may be avoided in someimplementations.

In some implementations, the filters can be used for not only pastmotion events, but also new motion events that have just occurred or arestill in progress. For example, when the video data of a detected motionevent candidate is processed, a live motion mask is created and updatedbased on each frame of the motion event as the frame is received by thevideo server system 508. In other words, after the live event mask isgenerated, it is updated as each new frame of the motion event isprocessed. In some implementations, the live event mask is compared tothe zone of interest on the fly, and as soon as a sufficient overlapfactor is accumulated, an alert is generated, and the motion event isidentified as an event of interest for the zone of interest. In someimplementations, an alert is presented on the review interface (e.g., asa pop-up) as the motion event is detected and categorized, and thereal-time alert optionally is formatted to indicate its associated zoneof interest (e.g., similar to the dialog box 928 in FIG. 9Ecorresponding to a motion event being associated with Event Category B).This provides real-time monitoring of the zone of interest in someimplementations.

In some implementations, the event mask of the motion event is generatedafter the motion event is completed, and the determination of theoverlap factor is based on a comparison of the completed event mask andthe zone of interest. Since the generation of the event mask issubstantially in real-time, real-time monitoring of the zone of interestmay also be realized this way in some implementations.

In some implementations, if multiple zones of interest are selected atany given time for a scene, the event mask of a new and/or old motionevent is compared to each of the selected zones of interest. For a newmotion event, if the overlap factor for any of the selected zones ofinterest exceeds the overlap threshold, an alert is generated for thenew motion event as an event of interest associated with the zone(s)that are triggered. For a previously stored motion event, if the overlapfactor for any of the selected zones of interest exceeds the overlapthreshold, the stored motion event is retrieved and presented to theuser as an event of interest associated with the zone(s) that aretriggered.

In some implementations, if a live event mask is used to monitor zonesof interest, a motion object in a motion event may enter different zonesat different times during the motion event. In some implementations, asingle alert (e.g., a pop-up notification over the timeline) isgenerated at the time that the motion event triggers a zone of interestfor the first time, and the alert can be optionally updated to indicatethe additional zones that are triggered when the live event mask touchesthose zones at later times during the motion event. In someimplementations, one alert is generated for each zone of interest whenthe live event mask of the motion event touches the zone of interest.

FIG. 11E illustrates an example process by which respective overlappingfactors are calculated for a motion event and several zones of interest.The zones of interest may be defined after the motion event has occurredand the event mask of the motion event has been stored, such as in thescenario of retrospective zone search. Alternatively, the zones ofinterest may also be defined before the motion event has occurred in thecontext of zone monitoring. In some implementations, zone monitoring canrely on a live event mask that is being updated as the motion event isin progress. In some implementations, zone monitoring relies on acompleted event mask that is formed immediately after the motion eventis completed.

As shown in the upper portion of FIG. 11E, motion masks 1118 for a framesequence of a motion event are generated as the motion event isprocessed for motion vector generation. Based on the motion masks 1118of the frames, an event mask 1120 is created. The creation of an eventmask based on motion masks has been discussed earlier with respect toFIG. 11C, and is not repeated herein.

Suppose that the motion masks 1118 shown in FIG. 11E are all the motionmasks of a past motion event, thus, the event mask 1120 is a completeevent mask stored for the motion event. After the event mask has beenstored, when a new zone of interest (e.g., Zone B among the selectedzones of interest 1122) is created later, the event mask 1120 iscompared to Zone B, and an overlap factor between the event mask 1120and Zone B is determined. In this particular example, Overlap B (withinOverlap 1124) is detected between the event mask 1120 and Zone B, and anoverlap factor based on Overlap B also exceeds an overlap threshold forqualifying the motion event as an event of interest for Zone B. As aresult, the motion event will be selectively retrieved and presented tothe reviewer, when the reviewer selects Zone B as a zone of interest fora present review session.

In some implementations, a zone of interest is created and selected forzone monitoring. During the zone monitoring, when a new motion event isprocessed in real-time, an event mask is created in real-time for thenew motion event and the event mask is compared to the selected zone ofinterest. For example, if Zone B is selected for zone monitoring, whenthe Overlap B is detected, an alert associated with Zone B is generatedand sent to the reviewer in real-time.

In some implementations, when a live event mask is used for zonemonitoring, the live event mask is updated with the motion mask of eachnew frame of a new motion event that has just been processed. The livemotion mask is compared to the selected zone(s) of interest 1122 atdifferent times (e.g., every 5 frames) during the motion event todetermine the overlap factor for each of the zones of interest. Forexample, if all of zones A, B, and C are selected for zone monitoring,at several times during the new motion event, the live event mask iscompared to the selected zones of interest 1122 to determine theircorresponding overlap factors. In this example, eventually, two overlapregions are found: Overlap A is an overlap between the event mask 1120and Zone A, and Overlap B is an overlap between the event mask 1120 andZone B. No overlap is found between the event mask 1120 and Zone C.Thus, the motion event is identified as an event of interest for bothZone A and Zone B, but not for Zone C. As a result, alerts will begenerated for the motion event for both Zone A and Zone B. In someimplementations, if the live event mask is compared to the selectedzones as the motion mask of each frame is added to the live event mask,Overlap A will be detected before Overlap B, and the alert for Zone Awill be triggered before the alert for Zone B.

It is noted that the motion event is detected and categorizedindependently of the existence of the zones of interest. In addition,the zone monitoring does not rely on raw image information within theselected zones; instead, the zone monitoring can take into account theraw image information from the entire scene. Specifically, the motioninformation during the entire motion event, rather than the motioninformation confined within the selected zone, is abstracted into anevent mask, before the event mask is used to determine whether themotion event is an event of interest for the selected zone. In otherwords, the context of the motion within the selected zones is preserved,and the event category of the motion event can be provided to the userto provide more meaning to the zone monitoring results.

Representative Processes

FIGS. 12A-12B illustrate a flowchart diagram of a method 1200 ofdisplaying indicators for motion events on an event timeline inaccordance with some implementations. In some implementations, themethod 1200 is performed by an electronic device with one or moreprocessors, memory, and a display. For example, in some implementations,the method 1200 is performed by client device 504 (FIGS. 5 and 7) or acomponent thereof (e.g., the client-side module 502, FIGS. 5 and 7). Insome implementations, the method 1200 is governed by instructions thatare stored in a non-transitory computer readable storage medium (e.g.,the memory 606, 706, or 806) and the instructions are executed by one ormore processors of the electronic device (e.g., the CPUs 512, 702, or802). Optional operations are indicated by dashed lines (e.g., boxeswith dashed-line borders).

In some implementations, control and access to the smart homeenvironment 100 is implemented in the operating environment 500 (FIG. 5)with a video server system 508 (FIGS. 5-6) and a client-side module 502(FIGS. 5 and 7) (e.g., an application for monitoring and controlling thesmart home environment 100) is executed on one or more client devices504 (FIGS. 5 and 7). In some implementations, the video server system508 manages, operates, and controls access to the smart home environment100. In some implementations, a respective client-side module 502 isassociated with a user account registered with the video server system508 that corresponds to a user of the client device 504.

The electronic device displays (1202) a video monitoring user interfaceon the display including a camera feed from a camera located remotelyfrom the client device in a first region of the video monitoring userinterface and an event timeline in a second region of the videomonitoring user interface, where the event timeline includes a pluralityof event indicators for a plurality of motion events previously detectedby the camera. In some implementations, the electronic device (i.e.,electronic device 166, FIG. 1, or client device 504, FIGS. 5 and 7) is amobile phone, tablet, laptop, desktop computer, or the like, whichexecutes a video monitoring application or program corresponding to thevideo monitoring user interface. In some implementations, the clientdevice 504 or a component thereof (e.g., event review interface module734, FIG. 7) displays the video monitoring user interface (UI) on thedisplay. FIG. 9C, for example, shows a video monitoring UI displayed bythe client device 504 with three distinct regions: a first region 903, asecond region 905, and a third region 907. In FIG. 9C, the first region903 of the video monitoring UI includes a video feed from a respectivecamera among the one or more camera 118 associated with the smart homeenvironment 100. In some implementations, the video feed is a live feedor playback of the recorded video feed from a previously selected startpoint. In FIG. 9C, the second region 905 of the video monitoring UIincludes an event timeline 910 and a current video feed indicator 909indicating the temporal position of the video feed displayed in thefirst region 903 (i.e., the point of playback for the video feeddisplayed in the first region 903). FIG. 9C, for example, shows eventindicators 922A, 922B, 922C, 922D, 922E, and 922F corresponding todetected motion events on the event timeline 910. In someimplementations, the video server system 508 or a component thereof(e.g., video data receiving module 616, FIG. 6) receives the video feedfrom the respective camera, and the video server system 508 or acomponent thereof (e.g., event detection module 620, FIG. 6) detects themotion events. In some implementations, the client device 504 receivesthe video feed either relayed through from the video server system 508or directly from the respective camera and detects the motion events.

In some implementations, at least one of the height or width of arespective event indicator among the plurality of event indicators onthe event timeline corresponds to (1204) the temporal length of a motionevent corresponding to the respective event indicator. In someimplementations, the event indicators can be no taller or wider than apredefined height/width so as not to clutter the event timeline. In FIG.9C, for example, the height of the indicators 922A, 922B, 922C, 922D,922E, and 922F indicate the temporal length of the motion events towhich they correspond.

In some implementations, the video monitoring user interface furtherincludes (1206) a third region with a list of one or more categories,and where the list of one or more categories at least includes an entrycorresponding to the first category after associating the first categorywith the first set of similar motion events. In some implementations,the first, second, and third regions are each located in distinct areasof the video monitoring interface. In some implementations, the list ofcategories includes recognized activity categories and created zones ofinterest. FIG. 9N, for example, shows the third region 907 of the videomonitoring UI with a list of categories for recognized event categoriesand created zones of interest. In FIG. 9N, the list of categories in thethird region 907 includes an entry 924A for a first recognized eventcategory labeled as “event category A,” an entry 924B for a secondrecognized event category labeled as “Birds in Flight,” and an entry924C for a previously created zone of interest labeled as “zone A.” Insome implementations, the list of categories in the third region 907also includes an entry for uncategorized motion events.

In some implementations, the entry corresponding to the first categoryincludes (1208) a text box for entering a label for the first category.In some implementations, events indicators on the event timeline arecolored according to the event category to which they are assigned andalso labeled with a text label corresponding to the event category towhich they are assigned. For example, in FIG. 9E, the entry 924A forevent category A and the entry 924B for event category B in the list ofcategories in the third region 907 of the video monitoring UI may eachfurther include a text box (not shown) for editing the default labelsfor the event categories. In this example, the user of the client device504 may edit the default labels for the event categories (e.g., “eventcategory A” and “event category B”) to a customized name (e.g.,“Coyotes” and “Birds in Flight”) using the corresponding text boxes.

In some implementations, the entry corresponding to the first categoryincludes (1210) a first affordance for disabling and enabling display ofthe first set of pre-existing event indicators on the event timeline. Insome implementations, the user of the client device is able to filterthe event timeline on a category basis (e.g., event categories and/orzones of interest) by disabling view of events indicators associatedwith unwanted categories. FIG. 9E, for example, shows an entry 924A forevent category A and an entry 924B for event category B in the list ofcategories in the third region 907 of the video monitoring UI. In FIG.9E, the entry 924A includes indicator filter 926A for enabling/disablingdisplay of event indicators on the event timeline 910 for motion eventsassigned to event category A, and the entry 924B includes indicatorfilter 926B for enabling/disabling display of event indicators on theevent timeline 910 for motion events assigned to event category B. InFIG. 9E, display of event indicators for motion events corresponding tothe event category A and the event category B are enabled as evinced bythe check marks corresponding to the indicator filter 926A and theindicator filter 926B. FIG. 9F, for example, shows the client device 504detecting a contact 930 (e.g., a tap gesture) at a locationcorresponding to the indicator filter 926A on the touch screen 906. FIG.9G, for example, shows the indicator filter 926A as unchecked inresponse to detecting the contact 930 in FIG. 9F. Moreover, in FIG. 9G,the client device 504 ceases to display event indicators 922A, 922C,922D, and 922E, which correspond to motion events assigned to eventcategory A, on the event timeline 910 in response to detecting thecontact 930 in FIG. 9F.

In some implementations, the entry corresponding to the first categoryincludes (1212) a second affordance for disabling and enablingnotifications corresponding to subsequent motion events of the firstcategory. In some implementations, the user of the client device is ableto disable reception of notifications for motion events that fall intocertain categories. FIG. 9E, for example, shows an entry 924A for eventcategory A and an entry 924B for event category B in the list ofcategories in the third region 907 of the video monitoring UI. In FIG.9E, the entry 924A includes notifications indicator 927A forenabling/disabling notifications sent in response to detection of motionevents assigned to event category A, and the entry 924B includesnotifications indicator 927B for enabling/disabling notifications sentin response to detection of motion events assigned to event category B.In FIG. 9E, notifications for detection of motion events correlated withevent category A and event category B are enabled. FIG. 9E, for example,also shows the client device 504 detecting a contact 929 (e.g., a tapgesture) at a location corresponding to the notifications indicator 927Aon the touch screen 906. FIG. 9F, for example, shows the notificationsindicator 927A in the third region 907 as disabled, shown by the linethrough the notifications indicator 927A, in response to detecting thecontact 929 in FIG. 9E.

In some implementations, the second region includes (1214) one or moretimeline length affordances for adjusting a resolution of the eventtimeline. In FIG. 9A, for example, the second region 905 includesaffordances 913 for changing the scale of event timeline 910: a 5 minuteaffordance 913A for changing the scale of the event timeline 910 to 5minutes, a 1 hour affordance 913B for changing the scale of the eventtimeline 910 to 1 hour, and a 24 hours affordance 913C for changing thescale of the event timeline 910 to 24 hours. In FIG. 9A, the scale ofthe event timeline 910 is 1 hour as evinced by the darkened bordersurrounding the 1 hour affordance 913B and also the temporal tick marksshown on the event timeline 910. In some implementations, the displayedportion of the event timeline may be changed by scrolling vialeft-to-right or right-to-left swipe gestures. In some implementations,the scale of the timeline may be increased (e.g., 1 hour to 24 hours)with a pinch-out gesture to display a greater temporal length ordecreased (e.g., 1 hour to 5 minutes) with a pinch-in gesture to displaya lesser temporal length.

In some implementations, an adjustment to the resolution of the timelinecauses the event timeline to automatically be repopulated with eventsindicators based on the selected granularity. FIG. 9U, for example,shows the client device 504 detecting a contact 978 at a locationcorresponding to the 24 hours affordance 913C on the touch screen 906.FIG. 9V, for example, shows the client device 504 displaying the eventtimeline 910 with a 24 hour scale in response to detecting selection ofthe 24 hours affordance 913C in FIG. 9U. In FIG. 9V, the 24 hours scaleis evinced by the darkened border surrounding the 24 hours affordance913C and also the temporal tick marks shown on the event timeline 910.For example, a first set of event indicators are displayed on the eventtimeline 910 in FIG. 9U in the 1 hour scale. Continuing with thisexample, in response to detecting selection of the 24 hours affordance913C in FIG. 9U, a second set of event indicators (at least partiallydistinct from the first set of event indicators) are displayed on theevent timeline 910 in FIG. 9V in the 24 hours scale.

The electronic device associates (1216) a newly created first categorywith a set of similar motion events (e.g., previously uncategorizedevents) from among the plurality of motion events previously detected bythe camera. In some implementations, the newly created category is arecognized event category or a newly created zone of interest. In someimplementations, the client device 504 (FIGS. 5 and 7), the video serversystem 508 (FIGS. 5-6) or a component thereof (e.g., eventcategorization module 622, FIG. 6), or a combination thereof determinesa first event category and identifies the set of similar motion eventswith motion characteristics matching the first event category. In someimplementations, the set of similar motion events match a predeterminedevent template or a learned event type corresponding to the first eventcategory. In some implementations, the client device 504 (FIGS. 5 and7), the video server system 508 (FIGS. 5-6) or a component thereof(e.g., zone monitoring module 630, FIG. 6), or a combination thereofidentifies the set of similar motion events that occurred at least inpart within a newly created zone of interest. For example, the set ofsimilar motion events touch or overlap the newly created zone ofinterest.

In some implementations, the video server system 508 provides anindication of the set of similar motion events assigned to the newlycreated first category, and, in response, the client device 504associates the set of similar motion events with the newly created firstcategory (i.e., by performing operation 1222 or associating the set ofsimilar motion events with the created first category in a localdatabase). In some implementations, the video server system 508 providesevent characteristics for the set of similar motion events assigned tothe newly created first category, and, in response, the client device504 associates the set of similar motion events with the newly createdfirst category (i.e., by performing operation 1222 or associating theset of similar motion events with the created first category in a localdatabase).

In some implementations, the newly created category corresponds to(1218) a newly recognized event category. In FIG. 9D, for example, thelist of categories in the third region 907 of the video monitoring UIincludes an entry 924A for newly recognized event category A. In FIG.9D, motion events correlated with event indicators 922A, 922C, 922D, and922E have been retroactively assigned to event category A as shown bythe changed display characteristic of event indicators 922A, 922C, 922D,and 922E (e.g., vertical stripes). For example, the motion eventscorrelated with the event indicators 922A, 922C, 922D, and 922E werepreviously uncategorized in FIG. 9C as shown by the unfilled displaycharacteristic for the event indicators 922A, 922C, 922D, and 922E.

In some implementations, the newly created category corresponds to(1220) a newly created zone of interest. FIG. 9N, for example, shows theclient device 504 displaying an entry 924C for newly created zone A inthe list of categories in the third region 907 in response to creatingthe zone of interest in FIGS. 9L-9M. In FIG. 9N, the motion eventcorrelated with event indicator 922M has been retroactively associatedwith zone A as shown by the changed display characteristic of the eventindicator 922M (e.g., the ‘X’ at the bottom of the event indicator922M). For example, the motion event correlated with the event indicator922M was previously uncategorized in FIG. 9M as shown by the unfilleddisplay characteristic for the event indicator 922M.

In response to associating the first category with the first set ofsimilar motion events, the electronic device changes (1222) at least onedisplay characteristic for a first set of pre-existing event indicatorsfrom among the plurality of event indicators on the event timeline thatcorrespond to the first category, where the first set of pre-existingevent indicators correspond to the set of similar motion events. Forexample, pre-existing uncategorized events indicators on the eventtimeline that correspond to events that fall into the first eventcategory are retroactively colored a specific color or displayed in aspecific shading pattern that corresponds to the first event category.In some implementations, the display characteristic is a fill color ofthe event indicator, a shading pattern of the event indicator, anicon/symbol overlaid on the event indicator, or the like. In FIG. 9D,for example, the event indicators 922A, 922C, 922D, and 922E includevertical stripes as compared to no fill in FIG. 9C. In FIG. 9N, forexample, the event indicator 922M includes an ‘X’ symbol overlaid on itsbottom region as compared to no fill or symbol(s) in FIG. 9M.

In some implementations, the set of similar motion events is (1224) afirst set of similar motion events, and the electronic device:associates a newly created second category with a second set of similarmotion events from among the plurality of motion events previouslydetected by the camera, where the second set of similar motion events isdistinct from the first set of similar motion events; and, in responseto associating the second category with the second set of similar motionevents, changes at least one display characteristic for a second set ofpre-existing event indicators from among the plurality of eventindicators on the event timeline that correspond to the second category,where the second set of pre-existing event indicators correspond to thesecond set of similar motion events. The second set of similar motionevents and the second set of pre-existing event indicators are distinctfrom the first set of similar motion events and the first set ofpre-existing event indicators. In FIG. 9E, for example, the list ofcategories in the third region 907 of the video monitoring UI includesan entry 924B for newly recognized event category B. In FIG. 9E, motionevents correlated with event indicators 922F, 922G, 922H, 922J, and 922Khave been retroactively assigned to event category B as shown by thechanged display characteristic of event indicators 922F, 922G, 922H,922J, and 922K (e.g., a diagonal shading pattern). For example, themotion events correlated with the event indicators 922F, 922G, 922H,922J, and 922K were previously uncategorized in FIGS. 9C-9D as shown bythe unfilled display characteristic for the event indicators 922F, 922G,922H, 922J, and 922K.

In some implementations, the electronic device detects (1226) a firstuser input at a location corresponding to a respective event indicatoron the event timeline and, in response to detecting the first userinput, displays preview of a motion event corresponding to therespective event indicator. For example, the user of the client device504 hovers over the respective events indicator with a mouse cursor ortaps the respective events indicator with his/her finger to display apop-up preview pane with a short video clip (e.g., approximately threeseconds) of the motion event that corresponds to the respective eventsindicator. FIG. 9G, for example, shows the client device 504 detecting acontact 931 (e.g., a tap gesture) at a location corresponding to eventindicator 922B on the touch screen 906. FIG. 9H, for example, shows theclient device 504 displaying a dialog box 923 for a respective motionevent correlated with the event indicator 922B in response to detectingselection of the event indicator 922B in FIG. 9G. In someimplementations, the dialog box 923 may be displayed in response tosliding or hovering over the event indicator 922B. In FIG. 9H, thedialog box 923 includes the time the respective motion event wasdetected (e.g., 11:37:40 am) and a preview 932 of the respective motionevent (e.g., a static image, a series of images, or a video clip).

In some implementations, if the event timeline is set to a temporallength of 24 hours and multiple motion events occurred within a shorttime period (e.g., 60, 300, 600, etc. seconds), the respective eventsindicator may be associated with the multiple motion events and thepop-up preview pane may concurrently display video clips of the multiplemotion event that corresponds to the respective events indicator. FIG.9V, for example, shows the client device 504 displaying the eventtimeline 910 with a 24 hour scale in response to detecting selection ofthe 24 hours affordance 913C in FIG. 9U. FIG. 9V, for example, alsoshows the client device 504 detecting a contact 980 (e.g., a tapgesture) at a location corresponding to an event indicator 979 on thetouch screen 906. FIG. 9W, for example, shows the client device 504displaying a dialog box 981 for respective motion events correlated withthe event indicator 979 in response to detecting selection of the eventindicator 979 in FIG. 9V. In some implementations, the dialog box 981may be displayed in response to sliding or hovering over the eventindicator 979. In FIG. 9W, the dialog box 981 includes the times atwhich the respective motion events were detected (e.g., 6:35:05 am,6:45:15 am, and 6:52:45 am). In FIG. 9W, the dialog box 981 alsoincludes previews 982A, 982B, and 982C of the respective motion events(e.g., a static image, a series of images, or a video clip).

It should be understood that the particular order in which theoperations in FIGS. 12A-12B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods and/or processesdescribed herein (e.g., the process 1000, and the methods 1300, 1400,1500, and 1600) are also applicable in an analogous manner to the method1200 described above with respect to FIGS. 12A-12B.

FIGS. 13A-13B illustrate a flowchart diagram of a method of editingevent categories in accordance with some implementations. In someimplementations, the method 1300 is performed by an electronic devicewith one or more processors, memory, and a display. For example, in someimplementations, the method 1300 is performed by client device 504(FIGS. 5 and 7) or a component thereof (e.g., the client-side module502, FIGS. 5 and 7). In some implementations, the method 1300 isgoverned by instructions that are stored in a non-transitory computerreadable storage medium (e.g., the memory 606, 706, or 806) and theinstructions are executed by one or more processors of the electronicdevice (e.g., the CPUs 512, 702, or 802). Optional operations areindicated by dashed lines (e.g., boxes with dashed-line borders).

In some implementations, control and access to the smart homeenvironment 100 is implemented in the operating environment 500 (FIG. 5)with a video server system 508 (FIGS. 5-6) and a client-side module 502(FIGS. 5 and 7) (e.g., an application for monitoring and controlling thesmart home environment 100) is executed on one or more client devices504 (FIGS. 5 and 7). In some implementations, the video server system508 manages, operates, and controls access to the smart home environment100. In some implementations, a respective client-side module 502 isassociated with a user account registered with the video server system508 that corresponds to a user of the client device 504.

The electronic device displays (1302) a video monitoring user interfaceon the display with a plurality of affordances associated one or morerecognized activities. In some implementations, the electronic device(i.e., electronic device 166, FIG. 1, or client device 504, FIGS. 5 and7) is a mobile phone, tablet, laptop, desktop computer, or the like,which executes a video monitoring application or program correspondingto the video monitoring user interface. In some implementations, theclient device 504 or a component thereof (e.g., event review interfacemodule 734, FIG. 7) displays the video monitoring user interface (UI) onthe display.

In some implementations, the video monitoring user interface includes(1304): (A) a first region with a video feed from a camera locatedremotely from the client device; (B) a second region with an eventtimeline, where the event timeline includes a plurality event indicatorscorresponding to motion events, and where at least a subset of theplurality of event indicators are associated with the respective eventcategory; and (C) a third region with a list of one or more recognizedevent categories. FIG. 9N, for example, shows a video monitoring UIdisplayed by the client device 504 with three distinct regions: a firstregion 903, a second region 905, and a third region 907. In FIG. 9N, thefirst region 903 of the video monitoring UI includes a video feed from arespective camera among the one or more camera 118 associated with thesmart home environment 100. In some implementations, the video feed is alive feed or playback of the recorded video feed from a previouslyselected start point. In FIG. 9N, the second region 905 of the videomonitoring UI includes an event timeline 910 and a current video feedindicator 909 indicating the temporal position of the video feeddisplayed in the first region 903 (i.e., the point of playback for thevideo feed displayed in the first region 903). FIG. 9N, for example,shows event indicators 922F, 922G, 922H, 922I, 922J, 922K, 922L, and922M corresponding to detected motion events on the event timeline 910.In some implementations, the video server system 508 (FIGS. 5-6)receives the video feed from the respective camera and detects themotion events. In some implementations, the client device 504 (FIGS. 5and 7) receives the video feed either relayed through from the videoserver system 508 or directly from the respective camera and detects themotion events. In FIG. 9N, the third region 907 of the video monitoringUI includes a list of categories for recognized event categories andcreated zones of interest.

In some implementations, the list of one or more recognized eventcategories includes (1306) the plurality of affordances, where each ofthe plurality of affordances correspond to a respective one of the oneor more recognized event categories. In FIG. 9N, the list of categoriesin the third region 907 includes an entry 924A for a first recognizedevent category labeled as “event category A,” an entry 924B for a secondrecognized event category labeled as “Birds in Flight,” and an entry924C for a created zone of interest labeled as “zone A.”

In some implementations, the respective affordance is displayed (1308)in response to performing a gesture with respect to one of the eventindicators. For example, the user hovers over one of the eventindicators on the event timeline to display a pop-up box including avideo clip of the motion event corresponding to the event indicators andan affordance for accessing the editing user interface corresponding tothe respective event category. FIG. 9G, for example, shows the clientdevice 504 detecting a contact 931 (e.g., a tap gesture) at a locationcorresponding to the event indicator 922B on the touch screen 906. FIG.9H, for example, shows the client device 504 displaying a dialog box 923for a respective motion event correlated with the event indicator 922Bin response to detecting selection of the event indicator 922B in FIG.9G. In some implementations, the dialog box 923 may be displayed inresponse to sliding or hovering over the event indicator 922B. In FIG.9H, the dialog box 923 includes an affordance 933, which, when activated(e.g., with a tap gesture), causes the client device 504 to display anediting UI for the event category to which the respective motion eventis assigned (if any).

The electronic device detects (1310) a user input selecting a respectiveaffordance from the plurality of affordances in the video monitoringuser interface, the respective affordance being associated with arespective event category of the one or more recognized eventcategories. FIG. 9H, for example, shows the client device 504 detectinga contact 934 (e.g., a tap gesture) at a location corresponding to theentry 924B for event category B on the touch screen 906.

In response to detecting the user input, the electronic device displays(1312) an editing user interface for the respective event category onthe display with a plurality of animated representations in a firstregion of the editing user interface, where the plurality of animatedrepresentations correspond to a plurality of previously captured motionevents assigned to the respective event category. In someimplementations, an animated representation (i.e., sprites) includesapproximately ten frames from a corresponding motion event. For example,the ten frames are the best frames illustrating the captured motionevent. FIG. 9I, for example, shows the client device 504 displaying anediting user interface (UI) for event category B in response todetecting selection of the entry 924B in FIG. 9H. In FIG. 9I, theediting user interface for event category B includes two distinctregions: a first region 935; and a second region 937. The first region935 of the editing UI includes representations 936 (sometimes alsoherein called “sprites”) of motion events assigned to event category B.In some implementations, each of the representations 936 is a series offrames or a video clip of a respective motion event assigned to eventcategory B. For example, in FIG. 9I, each of the representations 936corresponds to a motion event of a bird flying from left to right acrossthe field of view of the respective camera (e.g., a west to northeastdirection).

In some implementations, the editing user interface further includes(1314) a second region with a representation of a video feed from acamera located remotely from the client device. In FIG. 9I, the secondregion 937 of the editing UI includes a representation of the video feedfrom the respective camera with a linear motion vector 942 representingthe typical path of motion for motion events assigned event category B.In some implementations, the representation is a live video feed fromthe respective camera. In some implementations, the representation is astatic image corresponding to a recently captured frame from video feedof the respective camera.

In some implementations, the representation in the second regionincludes (1316) a linear motion vector overlaid on the video feed, wherethe linear motion vector corresponds to a typical motion path for theplurality of previously captured motion events assigned to therespective event category. In FIG. 9I, for example, a linear motionvector 942 representing the typical path of motion for motion eventsassigned event category B is overlaid on the representation of the videofeed in the second region 937 of the editing UI.

In some implementations, the first region of the editing user interfacefurther includes (1318) an affordance for disabling and enablingnotifications corresponding to subsequent motion events of therespective event category. In FIG. 9I, for example, the first region 935of the editing UI further includes a notifications indicator 940 forenabling/disabling notifications sent in response to detection of motionevents assigned to event category B.

In some implementations, the first region of the editing user interfacefurther includes (1320) a text box for entering a label for therespective event category. In FIG. 9I, for example, the first region 935of the editing UI further includes a label text entry box 939 forrenaming the label for the event category from the default name (“eventcategory B”) to a custom name. FIG. 9J, for example, shows the label forthe event category as “Birds in Flight” in the label text entry box 939as opposed to the default label—“event category B”—in FIG. 9I.

In some implementations, the electronic device detects (1322) one ormore subsequent user inputs selecting one or more animatedrepresentations in the first region of the editing user interface and,in response to detecting the one or more subsequent user inputs, sends amessage to a server indicating the one or more selected animatedrepresentations, where a set of previously captured motion eventscorresponding to the one or more selected animated representations aredisassociated with the respective event category. In someimplementations, the user of the client device 504 removes animatedrepresentations for motion events that are erroneously assigned to theevent category. In some implementations, the client device 504 sends amessage to the video server system 508 indicating the removed motionevents, and, subsequently, the video server system 508 or a componentthereof (e.g., event categorization module 622, FIG. 6) re-computes amodel or algorithm for the event category based on the removed motionevents.

In FIG. 9I, for example, each of the representations 936 is associatedwith a checkbox 941. In some implementations, when a respective checkbox941 is unchecked (e.g., with a tap gesture) the motion eventcorresponding to the respective checkbox 941 is removed from the eventcategory B and, in some circumstances, the event category B isre-computed based on the removed motion event. For example, thecheckboxes 941 enable the user of the client device 504 to remove motionevents incorrectly assigned to an event category so that similar motionevents are not assigned to the event category in the future. FIG. 9I,for example, shows the client device 504 detecting a contact 943 (e.g.,a tap gesture) at a location corresponding to the checkbox 941C on thetouch screen 906 and contact 944 (e.g., a tap gesture) at a locationcorresponding to the checkbox 941E on the touch screen 906. For example,the user of the client device 504 intends to remove the motion eventscorresponding to the representation 936C and the representation 936E asthey do not show a bird flying in a west to northeast direction. FIG.9J, for example, shows the checkbox 941C corresponding to the motionevent correlated with the event indicator 922L and the checkbox 941Ecorresponding to the motion event correlated with the event indicator922J as unchecked in response to detecting the contact 943 and thecontact 944, respectively, in FIG. 9I.

It should be understood that the particular order in which theoperations in FIGS. 13A-13B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods and/or processesdescribed herein (e.g., the process 1000, and the methods 1200, 1400,1500, and 1600) are also applicable in an analogous manner to the method1300 described above with respect to FIGS. 13A-13B.

FIGS. 14A-14B illustrate a flowchart diagram of a method ofautomatically categorizing a detected motion event in accordance withsome implementations. In some implementations, the method 1400 isperformed by a computing system (e.g., the client device 504, FIGS. 5and 7; the video server system 508, FIGS. 5-6; or a combination thereof)with one or more processors and memory. In some implementations, themethod 1400 is governed by instructions that are stored in anon-transitory computer readable storage medium (e.g., the memory 606,706, or 806) and the instructions are executed by one or more processorsof the computing system (e.g., the CPUs 512, 702, or 802). Optionaloperations are indicated by dashed lines (e.g., boxes with dashed-lineborders).

In some implementations, control and access to the smart homeenvironment 100 is implemented in the operating environment 500 (FIG. 5)with a video server system 508 (FIGS. 5-6) and a client-side module 502(FIGS. 5 and 7) (e.g., an application for monitoring and controlling thesmart home environment 100) is executed on one or more client devices504 (FIGS. 5 and 7). In some implementations, the video server system508 manages, operates, and controls access to the smart home environment100. In some implementations, a respective client-side module 502 isassociated with a user account registered with the video server system508 that corresponds to a user of the client device 504.

The computing system displays (1402) a video monitoring user interfaceon the display including a video feed from a camera located remotelyfrom the client device in a first region of the video monitoring userinterface and an event timeline in a second region of the videomonitoring user interface, where the event timeline includes one or moreevent indicators corresponding to one or more motion events previouslydetected by the camera. In some implementations, the client device 504or a component thereof (e.g., event review interface module 734, FIG. 7)displays the video monitoring user interface (UI) on the display. FIG.9C, for example, shows a video monitoring UI displayed by the clientdevice 504 with three distinct regions: a first region 903, a secondregion 905, and a third region 907. In FIG. 9C, the first region 903 ofthe video monitoring UI includes a video feed from a respective cameraamong the one or more camera 118 associated with the smart homeenvironment 100. In some implementations, the video feed is a live feedor playback of the recorded video feed from a previously selected startpoint. In FIG. 9C, the second region 905 of the video monitoring UIincludes an event timeline 910 and a current video feed indicator 909indicating the temporal position of the video feed displayed in thefirst region 903 (i.e., the point of playback for the video feeddisplayed in the first region 903). FIG. 9C, for example, shows eventindicators 922A, 922B, 922C, 922D, 922E, and 922F corresponding todetected motion events on the event timeline 910. In someimplementations, the video server system 508 receives the video feedfrom the respective camera and detects the motion events. In someimplementations, the client device 504 receives the video feed eitherrelayed through from the video server system 508 or directly from therespective camera and detects the motion events. FIG. 9N, for example,shows the third region 907 of the video monitoring UI with a list ofcategories for recognized event categories and created zones ofinterest. In FIG. 9N, the list of categories in the third region 907includes an entry 924A for a first recognized event category labeled as“event category A,” an entry 924B for a second recognized event categorylabeled as “Birds in Flight,” and an entry 924C for a created zone ofinterest labeled as “zone A.” In some implementations, the list ofcategories in the third region 907 also includes an entry foruncategorized motion events.

The computing system detects (1404) a motion event. In someimplementations, the client device 504 (FIGS. 5 and 7) receives thevideo feed either relayed through the video server system 508 ordirectly from the respective camera, and the client device 504 detectsthe respective motion event. In some implementations, the video serversystem 508 (FIGS. 5-6) receives the video feed from the respectivecamera, and the video server system 508 or a component thereof (e.g.,event detection module 620, FIG. 6) detects a respective motion eventpresent in the video feed. Subsequently, the video server system 508sends an indication of the motion event along with a correspondingmetadata, such as a timestamp for the detected motion event andcategorization information, to the client device 504 along with therelayed video feed from the respective camera. Continuing with thisexample, the client device 504 detects the motion event in response toreceiving the indication from the video server system 508.

The computing system determines (1406) one or more characteristics forthe motion event. For example, the one or more characteristics includethe motion direction, linear motion vector for the motion event, thetime of the motion event, the area in the field-of-view of therespective in which the motion event is detected, a face or itemrecognized in the captured motion event, and/or the like.

In accordance with a determination that the one or more determinedcharacteristics for the motion event satisfy one or more criteria for arespective category, the computing system (1408): assigns the motionevent to the respective category; and displays an indicator for thedetected motion event on the event timeline with a displaycharacteristic corresponding to the respective category. In someimplementations, the one or more criteria for the respective eventcategory include a set of event characteristics (e.g., motion vector,event time, model/cluster similarity, etc.), whereby the motion event isassigned to the event category if its determined characteristics match acertain number of event characteristics for the category. In someimplementations, the client device 504 (FIGS. 5 and 7), the video serversystem 508 (FIGS. 5-6) or a component thereof (e.g., eventcategorization module 622, FIG. 6), or a combination thereof assigns thedetected motion event to an event category. In some implementations, theevent category is a recognized event category or a previously createdzone of interest. In some implementations, the client device 504 or acomponent thereof (e.g., event review interface module 734, FIG. 7)displays an indicator for the detected motion event on the eventtimeline 910 with a display characteristic corresponding to therespective category. In FIG. 9E, for example, the client device 504detects a respective motion event and assigns the respective motionevent to event category B. Continuing with this example, in FIG. 9E, theclient device 504 displays event indicator 922L corresponding to therespective motion event with a display characteristic for event categoryB (e.g., the diagonal shading pattern).

In some implementations, the respective category corresponds to (1410) arecognized event category. In some implementations, the client device504, the video server system 508 (FIGS. 5-6) or a component thereof(e.g., event categorization module 622, FIG. 6), or a combinationthereof assigns the detected motion event with motion characteristicsmatching a respective event category to the respective event category.

In some implementations, the respective category corresponds to (1412) apreviously created zone of interest. In some implementations, the clientdevice 504, the video server system 508 (FIGS. 5-6) or a componentthereof (e.g., event categorization module 622, FIG. 6), or acombination thereof determines that the detected motion event touches oroverlaps at least part of a previously created zone of interest.

In some implementations, in accordance with a determination that the oneor more determined characteristics for the motion event satisfy the oneor more criteria for the respective category, the computing system or acomponent thereof (e.g., the notification module 738, FIG. 7) displays(1414) a notification indicating that the detected motion event has beenassigned to the respective category. FIG. 9E, for example, shows clientdevice 504 displaying a notification 928 for a newly detected respectivemotion event corresponding to event indicator 922L. For example, as therespective motion event is detected and assigned to event category B,event indicator 922L is displayed on the event timeline 910 with thedisplay characteristic for event category B (e.g., the diagonal shadingpattern). Continuing with this example, after or as the event indicator922L is displayed on the event timeline 910, notification 928 pops-upfrom the event indicator 922L. In FIG. 9E, the notification 928 notifiesthe user of the client device 504 that the motion event detected at12:32:52 pm was assigned to event category B.

In some implementations, the notification pops-up (1416) from theindicator for the detected motion event. In FIG. 9E, for example, thenotification 928 pops-up from the event indicator 922L after or as theevent indicator 922L is displayed on the event timeline 910.

In some implementations, the notification is overlaid (1418) on thevideo in the first region of the video monitoring user interface. Insome implementations, for example, the notification 928 in FIG. 9E is atleast partially overlaid on the video feed displayed in the first region903.

In some implementations, the notification is (1420) a bannernotification displayed in a location corresponding to the top of thevideo monitoring user interface. In some implementations, for example,the notification 928 in FIG. 9E pops-up from the event timeline 910 andis displayed at a location near the top of the first region 903 (e.g.,as a banner notification). In some implementations, for example, thenotification 928 in FIG. 9E pops-up from the event timeline 910 and isdisplayed in the center of the first region 903 (e.g., overlaid on thevideo feed).

In some implementations, the notification includes (1422) one or moreaffordances for providing feedback as to whether the detected motionevent is properly assigned to the respective category. In someimplementations, for example, the notification 928 in FIG. 9E includesone or more affordances (e.g., a thumbs up affordance and a thumbs downaffordance, or a properly categorized affordance and an improperlycategorized affordance) for providing feedback as to whether the motionevent correlated with event indicator 922L was properly assigned toevent category B.

It should be understood that the particular order in which theoperations in FIGS. 14A-14B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods and/or processesdescribed herein (e.g., the process 1000, and the methods 1200, 1300,1500, and 1600) are also applicable in an analogous manner to the method1400 described above with respect to FIGS. 14A-14B.

FIGS. 15A-15C illustrate a flowchart diagram of a method of generating asmart time-lapse video clip in accordance with some implementations. Insome implementations, the method 1500 is performed by an electronicdevice with one or more processors, memory, and a display. For example,in some implementations, the method 1500 is performed by client device504 (FIGS. 5 and 7) or a component thereof (e.g., the client-side module502, FIGS. 5 and 7). In some implementations, the method 1500 isgoverned by instructions that are stored in a non-transitory computerreadable storage medium (e.g., the memory 606, 706, or 806) and theinstructions are executed by one or more processors of the electronicdevice (e.g., the CPUs 512, 702, or 802). Optional operations areindicated by dashed lines (e.g., boxes with dashed-line borders).

In some implementations, control and access to the smart homeenvironment 100 is implemented in the operating environment 500 (FIG. 5)with a video server system 508 (FIGS. 5-6) and a client-side module 502(FIGS. 5 and 7) (e.g., an application for monitoring and controlling thesmart home environment 100) is executed on one or more client devices504 (FIGS. 5 and 7). In some implementations, the video server system508 manages, operates, and controls access to the smart home environment100. In some implementations, a respective client-side module 502 isassociated with a user account registered with the video server system508 that corresponds to a user of the client device 504.

The electronic device displays (1502) a video monitoring user interfaceon the display including a video feed from a camera located remotelyfrom the client device in a first region of the video monitoring userinterface and an event timeline in a second region of the videomonitoring user interface, where the event timeline includes a pluralityof event indicators for a plurality of motion events previously detectedby the camera. In some implementations, the electronic device (i.e.,electronic device 166, FIG. 1, or client device 504, FIGS. 5 and 7) is amobile phone, tablet, laptop, desktop computer, or the like, whichexecutes a video monitoring application or program corresponding to thevideo monitoring user interface. In some implementations, the clientdevice 504 or a component thereof (e.g., event review interface module734, FIG. 7) displays the video monitoring user interface (UI) on thedisplay. FIG. 9C, for example, shows a video monitoring UI displayed bythe client device 504 with three distinct regions: a first region 903, asecond region 905, and a third region 907. In FIG. 9C, the first region903 of the video monitoring UI includes a video feed from a respectivecamera among the one or more camera 118 associated with the smart homeenvironment 100. In some implementations, the video feed is a live feedor playback of the recorded video feed from a previously selected startpoint. In FIG. 9C, the second region 905 of the video monitoring UIincludes an event timeline 910 and a current video feed indicator 909indicating the temporal position of the video feed displayed in thefirst region 903 (i.e., the point of playback for the video feeddisplayed in the first region 903). FIG. 9C, for example, shows eventindicators 922A, 922B, 922C, 922D, 922E, and 922F corresponding todetected motion events on the event timeline 910. In someimplementations, the video server system 508 receives the video feedfrom the respective camera and detects the motion events. In someimplementations, the client device 504 receives the video feed eitherrelayed through from the video server system 508 or directly from therespective camera and detects the motion events. FIG. 9N, for example,shows the third region 907 of the video monitoring UI with a list ofcategories for recognized event categories and created zones ofinterest. In FIG. 9N, the list of categories in the third region 907includes an entry 924A for a first recognized event category labeled as“event category A,” an entry 924B for a second recognized event categorylabeled as “Birds in Flight,” and an entry 924C for a created zone ofinterest labeled as “zone A.” In some implementations, the list ofcategories in the third region 907 also includes an entry foruncategorized motion events.

The electronic device detects (1504) a first user input selecting aportion of the event timeline, where the selected portion of the eventtimeline includes a subset of the plurality of event indicators on theevent timeline. For example, the user of the client device selects theportion of the event timeline by inputting a start and end time or usinga sliding, adjustable window overlaid on the timeline. In FIG. 9O, forexample, the second region 905 of the video monitoring UI includes astart time entry box 956A for entering/changing a start time of thetime-lapse video clip to be generated and an end time entry box 956B forentering/changing an end time of the time-lapse video clip to begenerated. In FIG. 9O, the second region 905 of the video monitoring UIalso includes a start time indicator 957A and an end time indicator 957Bon the event timeline 910, which indicates the start and end times ofthe time-lapse video clip to be generated. In some implementations, forexample, the locations of the start time indicator 957A and the end timeindicator 957B in FIG. 9O may be moved on the event timeline 910 viapulling/dragging gestures.

In response to the first user input, the electronic device causes (1506)generation of a time-lapse video clip of the selected portion of theevent timeline. In some implementations, after selecting the portion ofthe event timeline, the client device 504 causes generation of thetime-lapse video clip corresponding to the selected portion by theclient device 504, the video server system 508 or a component thereof(e.g., event post-processing module 634, FIG. 6), or a combinationthereof. In some implementations, the motion events within the selectedportion of the event timeline are played at a slower speed than thebalance of the selected portion of the event timeline. In someimplementations, the motion events assigned to enabled event categoriesand motion events that touch or overlap enabled zones are played at aslower speed than the balance of the selected portion of the eventtimeline including motion events assigned to disabled event categoriesand motion events that touch or overlap disabled zones.

In some implementations, prior to detecting the first user inputselecting the portion of the event timeline, the electronic device(1508): detects a third user input selecting a time-lapse affordancewithin the video monitoring user interface; and, in response todetecting the third user input, displays at least one of (A) anadjustable window overlaid on the event timeline for selecting theportion of the event timeline and (B) one or more text entry boxes forentering times for a beginning and an end of the portion of the eventtimeline. In some implementations, the first user input corresponds tothe adjustable window or the one or more text entry boxes. In FIG. 9N,for example, the second region 905 includes “Make Time-Lapse” affordance915, which, when activated (e.g., via a tap gesture), enables the userof the client device 504 to select a portion of the event timeline 910for generation of a time-lapse video clip (as shown in FIGS. 9N-9Q).FIG. 9N, for example, shows the client device 504 detecting a contact954 (e.g., a tap gesture) at a location corresponding to the “MakeTime-Lapse” affordance 915 on the touch screen 906. For example, thecontact 954 is the third user input. FIG. 9O, for example, shows theclient device 504 displaying controls for generating a time-lapse videoclip in response to detecting selection of the “Make Time-Lapse”affordance 915 in FIG. 9N. In FIG. 9O, the second region 905 of thevideo monitoring UI includes a start time entry box 956A forentering/changing a start time of the time-lapse video clip to begenerated and an end time entry box 956B for entering/changing an endtime of the time-lapse video clip to be generated. In FIG. 9O, thesecond region 905 also includes a start time indicator 957A and an endtime indicator 957B on the event timeline 910, which indicates the startand end times of an adjustable window on the event timeline 910corresponding to the time-lapse video clip to be generated. In someimplementations, for example, the locations of the start time indicator957A and the end time indicator 957B in FIG. 9O may be moved on theevent timeline 910 via dragging gestures.

In some implementations, causing generation of the time-lapse video clipfurther comprises (1510) sending an indication of the selected portionof the event timeline to a server so as to generate the time-lapse videoclip of the selected portion of the event timeline. In someimplementations, after detecting the first user input selecting theportion of the event timeline, the client device 504 causes thetime-lapse video clip to be generated by sending an indication of thestart time (e.g., 12:20:00 pm according to the start time entry box 956Ain FIG. 9O) and the end time (e.g., 12:42:30 pm according to the endtime entry box 956B in FIG. 9O) of the selected portion to the videoserver system 508. Subsequently, in some implementations, the videoserver system 508 or a component thereof (e.g., event post-processingmodule 643, FIG. 6) generates the time-lapse video clip according to theindication of the start time and the end time and detected motion eventsthat fall between the start time and the end time.

In some implementations, causing generation of the time-lapse video clipfurther comprises (1512) generating the time-lapse video clip fromstored video footage based on the selected portion of the event timelineand timing of the motion events corresponding to the subset of theplurality of event indicators within the selected portion of the eventtimeline. In some implementations, after detecting the first user inputselecting the portion of the event timeline, the client device 504generates the time-lapse video clip from stored footage according to thestart time (e.g., 12:20:00 pm according to the start time entry box 956Ain FIG. 9O) and the end time (e.g., 12:42:30 pm according to the endtime entry box 956B in FIG. 9O) indicated by the user of the clientdevice 504 and detected motion events that fall between the start timeand the end time. In some implementations, the client device generatesthe time-lapse video clip by modifying the playback speed of the storedfootage based on the timing of motion events instead of generating a newvideo clip from the stored footage.

In some implementations, causing generation of the time-lapse video clipfurther comprises (1514) detecting a third user input selecting atemporal length for the time-lapse video clip. In some implementations,prior to generation of the time-lapse video clip and after selecting theportion of the event timeline, the client device 504 displays a dialogbox or menu pane that enables the user of the client device 504 toselect a length of the time-lapse video clip (e.g., 30, 60, 90, etc.seconds). For example, the user selects a two hour portion of the eventtimeline for the time-lapse video clip and then selects a 60 secondlength for the time-lapse video clip which causes the selected 2 hourportion of the event timeline to be compressed to 60 seconds in length.

In some implementations, after causing generation of the time-lapsevideo clip, the electronic device displays (1516) a first notificationwithin the video monitoring user interface indicating processing of thetime-lapse video clip. For example, the first notification is a bannernotification indicating the time left in generating/processing of thetime-lapse video clip. FIG. 9P, for example, shows client device 504displaying a notification 961 overlaid on the first region 903 (e.g., abanner notification). In FIG. 9P, the notification 961 indicates thatthe time-lapse video clip is being processed and also includes an exitaffordance 962, which, when activated (e.g., with a tap gesture), causesthe client device 504 the client device 504 to dismiss the notification961.

The electronic device displays (1518) the time-lapse video clip of theselected portion of the event timeline, where motion eventscorresponding to the subset of the plurality of event indicators areplayed at a slower speed than the remainder of the selected portion ofthe event timeline. For example, during playback of the time-lapse videoclip, motion events are displayed at 2× or 4× speed and other portionsof the video feed within the selection portion are displayed at 16× or32× speed.

In some implementations, prior to displaying the time-lapse video clip,the electronic device (1520): displays a second notification within thevideo monitoring user interface indicating completion of generation forthe time-lapse video clip; and detects a fourth user input selecting thesecond notification. In some implementations, displaying the time-lapsevideo clip further comprises displaying the time-lapse video clip inresponse to detecting the fourth input. For example, the secondnotification is a banner notification indicating that generation of thetime-lapse video clip is complete. At a time subsequent to FIG. 9P, thenotification 961 in FIG. 9Q indicates that processing of the time-lapsevideo clip is complete and includes a “Play Time-Lapse” affordance 963,which, when activated (e.g., with a tap gesture), causes the clientdevice 504 to play the time-lapse video clip.

In some implementations, prior to displaying the time-lapse video clip,the electronic device detects (1522) selection of the time-lapse videoclip from a collection of saved video clips. In some implementations,displaying the time-lapse video clip further comprises displaying thetime-lapse video clip in response to detecting selection of thetime-lapse video clip. In some implementations, the server video serversystem 508 stores a collection of saved video clips (e.g., in the videostorage database 516, FIGS. 5-6) including time-lapse video clips andnon-time-lapse videos clips. In some implementations, the user of theclient device 504 is able to access and view the saved clips at anytime.

In some implementations, the electronic device detects (1524) one ormore second user inputs selecting one or more categories associated withthe plurality of motion events. In some implementations, causinggeneration of the time-lapse video clip further comprises causinggeneration of the time-lapse video clip of the selected portion of theevent timeline based on the one or more selected categories, anddisplaying the time-lapse video clip further comprises displaying thetime-lapse video clip of the selected portion of the event timeline,where motion events corresponding to the subset of the plurality ofevent indicators assigned to the one or more selected categories areplayed at a slower speed than the remainder of the selected portion ofthe event timeline. In some implementations, the one or more selectedcategories include (1526) at least one of a recognized event category ora previously created zone of interest. In some implementations, the userof the client device 504 is able to enable/disable zones and/or eventcategories prior to generating the time-lapse video clip. For example,the motion events assigned to enabled event categories and motion eventsthat touch or overlap enabled zones are played at a slower speed duringthe time-lapse than the balance of the selected portion of the eventtimeline including motion events assigned to disabled event categoriesand motion events that touch or overlap disabled zones.

In FIG. 9O, for example, the list of categories in the third region 907of the video monitoring UI includes entries for three categories: afirst entry 924A corresponding to event category A; a second entry 924Bcorresponding to the “Birds in Flight” event category; and a third entry924C corresponding to zone A (e.g., created in FIGS. 9L-9M). Each of theentries 924 includes an indicator filter 926 for enabling/disablingmotion events assigned to the corresponding category. In FIG. 9O, forexample, indicator filter 924A in the entry 924A corresponding to eventcategory A is disabled, indicator filter 924B in the entry 924Bcorresponding to the “Birds in Flight” event category is enabled, andindicator filter 924C in the entry 924C corresponding to zone A isenabled. Thus, for example, after detecting a contact 955 at a locationcorresponding to the “Create Time-Lapse” affordance 958 on the touchscreen 906 in FIG. 9O, the client device 504 causes generation of atime-lapse video clip according to the selected portion of the eventtimeline 910 (i.e., the portion corresponding to the start and end timesdisplayed by the start time entry box 956A and the end time entry box956B) and the enabled categories. For example, motion events assigned tothe “Birds in Flight” event category and motion events overlapping ortouching zone A will be played at 2× or 4× speed and the balance of theselected portion (including motion events assigned to event category A)will be displayed at 16× or 32× speed during playback of the time-lapsevideo clip.

It should be understood that the particular order in which theoperations in FIGS. 15A-15C have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods and/or processesdescribed herein (e.g., the process 1000, and the methods 1200, 1300,1400, and 1600) are also applicable in an analogous manner to the method1500 described above with respect to FIGS. 15A-15C.

FIGS. 16A-16B illustrate a flowchart diagram of a method of performingclient-side zooming of a remote video feed in accordance with someimplementations. In some implementations, the method 1600 is performedby an electronic device with one or more processors, memory, and adisplay. For example, in some implementations, the method 1600 isperformed by client device 504 (FIGS. 5 and 7) or a component thereof(e.g., the client-side module 502, FIGS. 5 and 7). In someimplementations, the method 1600 is governed by instructions that arestored in a non-transitory computer readable storage medium (e.g., thememory 606, 706, or 806) and the instructions are executed by one ormore processors of the electronic device (e.g., the CPUs 512, 702, or802). Optional operations are indicated by dashed lines (e.g., boxeswith dashed-line borders).

In some implementations, control and access to the smart homeenvironment 100 is implemented in the operating environment 500 (FIG. 5)with a video server system 508 (FIGS. 5-6) and a client-side module 502(FIGS. 5 and 7) (e.g., an application for monitoring and controlling thesmart home environment 100) is executed on one or more client devices504 (FIGS. 5 and 7). In some implementations, the video server system508 manages, operates, and controls access to the smart home environment100. In some implementations, a respective client-side module 502 isassociated with a user account registered with the video server system508 that corresponds to a user of the client device 504.

The electronic device receives (1602) a first video feed from a cameralocated remotely from the client device with a first field of view. Insome implementations, the electronic device (i.e., electronic device166, FIG. 1, or client device 504, FIGS. 5 and 7) is a mobile phone,tablet, laptop, desktop computer, or the like, which executes a videomonitoring application or program corresponding to the video monitoringuser interface. In some implementations, the video feed from therespective camera is relayed to the client device 504 by the videoserver system 508. In some implementations, the client device 504directly receives the video feed from the respective camera.

The electronic device displays (1604), on the display, the first videofeed in a video monitoring user interface. In some implementations, theclient device 504 or a component thereof (e.g., event review interfacemodule 734, FIG. 7) displays the video monitoring user interface (UI) onthe display. FIG. 9C, for example, shows a video monitoring UI displayedby the client device 504 with three distinct regions: a first region903, a second region 905, and a third region 907. In FIG. 9C, the firstregion 903 of the video monitoring UI includes a video feed from arespective camera among the one or more camera 118 associated with thesmart home environment 100. In some implementations, the video feed is alive feed or playback of the recorded video feed from a previouslyselected start point. In FIG. 9C, for example, an indicator 912indicates that the video feed being displayed in the first region 903 isa live video feed.

The electronic device detects (1606) a first user input to zoom in on arespective portion of the first video feed. In some implementations, thefirst user input is a mouse scroll wheel, keyboard shortcuts, orselection of a zoom-in affordance (e.g., elevator bar or other widget)in a web browser accompanied by a dragging gesture to pane the zoomedregion. For example, the user of the client device 504 is able to dragthe handle 919 of the elevator bar in FIG. 9B to zoom-in on the videofeed. Subsequently, the user of the client device 504 may perform adragging gesture inside of the first region 903 to pane up, down, left,right, or a combination thereof.

In some implementations, the display is (1608) a touch-screen display,and where the first user input is a pinch-in gesture performed on thefirst video feed within the video monitoring user interface. In someimplementations, the first user input is a pinch-in gesture on a touchscreen of the electronic device. FIG. 9R, for example, shows the clientdevice 504 detecting a pinch-in gesture with contacts 965A and 965Brelative to a respective portion of the video feed in the first region903 on the touch screen 906. In this example, the first user input isthe pinch-in gesture with contacts 965A and 965B.

In response to detecting the first user input, the electronic deviceperforms (1610) a software zoom function on the respective portion ofthe first video feed to display the respective portion of the firstvideo feed in a first resolution. In some implementations, the firstuser input determines a zoom magnification for the software zoomfunction. For example, the width between contacts of a pinch gesturedetermines the zoom magnification. In another example, the length of adragging gesture on an elevator bar associated with zooming determinesthe zoom magnification. FIG. 9S, for example, shows the client device504 displaying a zoomed-in portion of the video feed in response todetecting the pinch-in gesture on the touch screen 906 in FIG. 9R. Insome implementations, the zoomed-in portion of the video feedcorresponds to a software-based zoom performed locally by the clientdevice 504 on the respective portion of the video feed corresponding tothe pinch-in gesture in FIG. 9R.

In some implementations, in response to detecting the first user input,the electronic device displays (1612) a perspective window within thevideo monitoring user interface indicating a location of the respectiveportion relative to the first video feed. In some implementations, afterperforming the software zoom, a perspective window is displayed in thevideo monitoring UI which shows the zoomed region's location relative tothe first video feed (e.g., picture-in-picture window). FIG. 9S, forexample, shows the client device 504 displaying a perspective box 969 inthe first region 903, which indicates the zoomed-in portion 970 relativeto the full field of view of the respective camera.

In some implementations, prior to the determining and the sending, theelectronic device detects (1614) a second user input within the videomonitoring user interface selecting a video enhancement affordance. Insome implementations, the determining operation 1618 and the sendingoperation 1620 are performed in response to detecting the second userinput. In FIG. 9S, for example, the video controls in the first region903 of the video monitoring UI further includes an enhancementaffordance 968 in response to detecting the pinch-in gesture in FIG. 9R.When activated (e.g., with a tap gesture), the enhancement affordance968 causes the client device 504 to send a zoom command to therespective camera. In some implementations, the enhancement affordanceis only displayed to users with administrative privileges because itchanges the field of view of the respective camera and consequently therecorded video footage. FIG. 9S, for example, shows the client device504 detecting a contact 967 at a location corresponding to theenhancement affordance 968 on the touch screen 906.

In some implementations, in response to detecting the second user inputand prior to performing the sending operation 1620, the electronicdevice displays (1616) a warning message indicating that saved videofootage will be limited to the respective portion. In someimplementations, after selecting the enhancement affordance to hardwarezoom in on the respective portion, only footage from the respectiveportion (i.e., the cropped region) will be saved by the video serversystem 508. Prior to selecting the enhancement affordance, the videoserver system 508 saved the entire field of view of the respectivecamera shown in the first video feed, not the software zoomed version.FIG. 9T, for example, shows the client device 504 displaying a dialogbox 971 in response to detecting selection of the enhancement affordance968 in FIG. 9S. In FIG. 9T, the dialog box 971 warns the user of theclient device 504 that enhancement of the video feed will cause changesto the recorded video footage and also any created zones of interest. InFIG. 9T, the dialog box 971 includes: a cancel affordance 972, which,when activated (e.g., with a tap gesture) causes the client device 504to cancel of the enhancement operation and consequently cancel sendingof the zoom command; and an enhance affordance 973, when activated(e.g., with a tap gesture) causes the client device 504 to send the zoomcommand to the respective camera.

The electronic device determines (1618) a current zoom magnification ofthe software zoom function and coordinates of the respective portion ofthe first video feed. In some implementations, the client device 504 ora component thereof (e.g., camera control module 732, FIG. 7) determinesthe current zoom magnification of the software zoom function andcoordinates of the respective portion of the first video feed. Forexample, the coordinates are an offset from the center of the originalvideo feed to the center of the respective portion.

The electronic device sends (1620) a command to the camera to perform ahardware zoom function on the respective portion according to thecurrent zoom magnification and the coordinates of the respective portionof the first video feed. In some implementations, the client device 504or a component thereof (e.g., camera control module 732, FIG. 7) causesthe command to be sent to the respective camera, where the commandincludes the current zoom magnification of the software zoom functionand coordinates of the respective portion of the first video feed. Insome implementations, the command is typically relayed through the videoserver system 508 to the respective camera. In some implementations,however, the client device 504 sends the command directly to therespective camera. In some implementations, the command also changes theexposure of the respective camera and the focus point of directionalmicrophones of the respective camera. In some implementations, the videoserver system 508 stores video settings for the respective camera (e.g.,tilt, pan, and zoom settings) and the coordinates of the respectiveportion (i.e., the cropped region).

The electronic device receives (1622) a second video feed from thecamera with a second field of view different from the first field ofview, where the second field of view corresponds to the respectiveportion. For example, the second video feed is a cropped version of thefirst video feed that only includes the respective portion in itsfield-of-view, but with higher resolution than the local software zoomedversion of the respective portion.

The electronic device displays (1624), on the display, the second videofeed in the video monitoring user interface, where the second video feedis displayed in a second resolution that is higher than the firstresolution. FIG. 9U, for example, shows the client device 504 displayingthe zoomed-in portion of the video feed at a higher resolution ascompared to FIG. 9S in response to detecting selection of the enhanceaffordance 973 in FIG. 9T. In some implementations, a scene changedetector associated with the application resets the local, software zoomwhen the total pixel color difference between a frame from the secondvideo feed and a previous frame from the first video feed exceeds apredefined threshold. In some implementations, the user may perform asecond software zoom and enhancement zoom operation. In someimplementations, the video monitoring user interface indicates thecurrent zoom magnification of the software and/or hardware zoom. Forexample, the video monitoring UI in FIG. 9S further indicates thecurrent zoom magnification in text (e.g., overlaid on the first region903). In some implementations, the total combined zoom magnification maybe limited to a predetermined zoom magnification (e.g., 8×). In someimplementations, the user may zoom & enhance multiple different regionsof the first video feed for concurrent display in the video monitoringinterface. For example, each of the regions is displayed in its ownsub-region in the first region 903 of the video monitoring interfacewhile the live video feed from the respective camera is displayed in thefirst region 903.

In some implementations, the video monitoring user interface includes(1626) an affordance for resetting the camera to display the first videofeed after displaying the second video feed. In some implementations,after performing the hardware zoom, the user of the client device 504 isable to reset the zoom configuration to the original video feed. In FIG.9U, for example, the video controls in the first region 903 of the videomonitoring UI further include a zoom reset affordance 975, which, whenactivated (e.g., with a tap gesture) causes the client device 504 resetthe zoom magnification of the video feed to its original setting (e.g.,as in FIG. 9R prior to the pinch-in gesture).

It should be understood that the particular order in which theoperations in FIGS. 16A-16B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods and/or processesdescribed herein (e.g., the process 1000, and the methods 1200, 1300,and 1500) are also applicable in an analogous manner to the method 1600described above with respect to FIGS. 16A-16B.

FIGS. 17A-17D illustrate a flowchart diagram of a method 1700 ofprocessing data for video monitoring on a computing system (e.g., thecamera 118, FIGS. 5 and 8; a controller device; the video server system508, FIGS. 5-6; or a combination thereof) in accordance with someimplementations. FIGS. 17A-17D correspond to instructions stored in acomputer memory or computer readable storage medium (e.g., the memory606, 706, or 806).

In this representative method, the start of a motion event candidate isdetected in a live video stream, which then triggers the subsequentprocessing (e.g., motion track and motion vector generation) andcategorization of the motion event candidate. A simple spatial motionvector, such as a linear motion vector is optionally used to representthe motion event candidate in the event categorization process toimprove processing efficiency (e.g., speed and data compactness).

As shown in FIG. 17A, the method is performed at a computing systemhaving one or more processors and memory. In some implementations, thecomputing system may be the camera 118, the controller device, thecombination of the camera 118 and the controller device, the combinationof video source 522 (FIG. 5) and the event preparer of the video serversystem 508, or the combination of the video source 522 and the videoserver system 508. The implementation optionally varies depending on thecapabilities of the various sub-systems involved in the data processingpipeline as shown in FIG. 11A.

The computing system processes (1702) the video stream to detect a startof a first motion event candidate in the video stream. In response todetecting the start of the first motion event candidate in the videostream, the computing system initiates (1704) event recognitionprocessing on a first video segment associated with the start of thefirst motion event candidate, where initiating the event recognitionprocessing further includes the following operations: determining amotion track of a first object identified in the first video segment;generating a representative motion vector for the first motion eventcandidate based on the respective motion track of the first object; andsending the representative motion vector for the first motion eventcandidate to an event categorizer, where the event categorizer assigns arespective motion event category to the first motion event candidatebased on the representative motion vector of the first motion eventcandidate.

In some implementations, at least one of processing the video stream,determining the motion track, generating the representative motionvector, and sending the representative motion vector to the eventcategorizer is (1706) performed locally at the source of the videostream. For example, in some implementations, the camera 118 may performone or more of the initial tasks locally before sending the rest of thetasks to the cloud for the server to complete. In some implementations,all of the above tasks are performed locally at the camera 118 or thevideo source 522 comprising the camera 118 and a controller device.

In some implementations, at least one of processing the video stream,determining the motion track, generating the representative motionvector, and sending the representative motion vector to thecategorization server is (1708) performed at a server (e.g., the videoserver system 508) remote from the source of the video stream (e.g.,video source 522). In some implementations, all of the above tasks areperformed at the server, and the video source is only responsible forstreaming the video to the server over the one or more networks 162(e.g., the Internet).

In some implementations, the computing system includes (1710) at leastthe source of the video stream (e.g., the video source 522) and a remoteserver (e.g., the video server system 508), and the source of the videostream dynamically determines whether to locally perform the processingof the video stream, the determining of the motion track, and thegenerating of the representative motion vector, based on one or morepredetermined distributed processing criteria. For example, in someimplementations, the camera dynamically determines how to divide up theabove tasks based on the current network conditions, the localprocessing power, the number and frequency of motion events that areoccurring right now or on average, the current load on the server, thetime of day, etc.

In some implementations, in response to detecting the start of the firstmotion event candidate, the computing system (e.g., the video source522) uploads (1712) the first video segment from the source of the videostream to a remote server (e.g., the video server system 508), where thefirst video segment begins at a predetermined lead time (e.g., 5seconds) before the start of the first motion event candidate and lastsa predetermined duration (e.g., 30 seconds). In some implementations,the uploading of the first video segment is in addition to the regularvideo stream uploaded to the video server system 508.

In some implementations, when uploading the first video segment from thesource of the video stream to the remote server: the computing system(e.g., the video source 522), in response to detecting the start of thefirst motion event candidate, uploads (1714) the first video segment ata higher quality level as compared to a normal quality level at whichvideo data is uploaded for cloud storage. For example, in someimplementations, a high resolution video segment is uploaded for motionevent candidates detected in the video stream, so that the video segmentcan be processed in various ways (e.g., zoomed, analyzed, filtered byzones, filtered by object types, etc.) in the future. Similarly, in someimplementations, the frame rate of the video segment for detected eventcandidate is higher that the video data uploaded for cloud storage.

In some implementations, in response to detecting the start of the firstmotion event candidate, the computing system (e.g., the event preparerof the video server system 508) extracts (1716) the first video segmentfrom cloud storage (e.g., video data database 1106, FIG. 11A) for thevideo stream, where the first video segment begins at a predeterminedlead time (e.g., 5 seconds) before the start of the first motion eventcandidate and lasts a predetermined duration (e.g., 30 seconds).

In some implementations, to process the video stream to detect the startof the first motion event candidate in the video stream: the computingsystem performs (1718) the following operations: obtaining a profile ofmotion pixel counts for a current frame sequence in the video stream; inresponse to determining that the obtained profile of motion pixel countsmeet a predetermined trigger criterion (e.g., total motion pixel countexceeds a predetermined threshold), determining that the current framesequence includes a motion event candidate; identifying a beginning timefor a portion of the profile meeting the predetermined triggercriterion; and designating the identified beginning time to be the startof the first motion event candidate. This is part of the processingpipeline 1104 (FIG. 11A) for detecting a cue point, which may beperformed locally at the video source 522 (e.g., by the camera 118). Insome implementations, the profile is a histogram of motion pixel countat each pixel location in the scene depicted in the video stream. Moredetails of cue point detection are provided earlier in FIG. 11A andaccompanying descriptions.

In some implementations, the computing system receives (1720) arespective motion pixel count for each frame of the video stream from asource of the video stream. In some implementations, the respectivemotion pixel count is adjusted (1722) for one or more of changes ofcamera states during generation of the video stream. For example, insome implementations, the adjustment based on camera change (e.g.,suppressing the motion event candidate altogether if the cue pointoverlaps with a camera state change) is part of the false positivesuppression process performed by the video source. The changes in camerastates include camera events such as IR mode change or AE change, and/orcamera system reset.

In some implementations, to obtain the profile of motion pixel countsfor the current frame sequence in the video stream, the computing systemperforms (1724) the following operations: generating a raw profile basedon the respective motion pixel count for each frame in the current framesequence; and generating the profile of motion pixel counts by smoothingthe raw profile to remove one or more temporary dips in pixel counts inthe raw profile. This is illustrated in FIG. 11B-(b) and accompanyingdescriptions.

In some implementations, to determine the motion track of the objectidentified in the first video segment, the computing system performs(1726) the following operations: based on a frame sequence of the firstvideo segment: (1) performing background estimation to obtain abackground for the first video segment; (2) performing objectsegmentation to identify one or more foreground objects in the firstvideo segment by subtracting the obtained background from the framesequence, the one or more foreground object including the object; and(3) establishing a respective motion track for each of the one or moreforeground objects by associating respective motion masks of theforeground object across multiple frames of the frame sequence. Themotion track generation is described in more detail in FIG. 11A andaccompanying descriptions.

In some implementations, the computing system determines (1728) aduration of the respective motion track for each of the one or moreforeground objects, discards (1730) zero or more respective motiontracks and corresponding foreground objects if the durations of therespective zero or more motion tracks are shorter than a predeterminedduration (e.g., 8 frames). This is optionally included as part of thefalse positive suppression process. Suppression of super short trackshelps to prune off movements such as leaves in a tree, etc.

In some implementations, to perform the object segmentation to identifyone or more foreground objects and establish the respective motion trackfor each of the one or more foreground objects, the computing systemperforms (1732) the following operations: building a histogram offoreground pixels identified in the frame sequence of the first videosegment, where the histogram specifies a frame count for each pixellocation in a scene of the first video segment; filtering the histogramto remove regions below a predetermined frame count; segmenting thefiltered histogram into the one or more motion regions; and selectingone or more dominant motion regions from the one or more motion regionsbased on a predetermined dominance criterion (e.g., regions containingat least a threshold of frame count/total motion pixel count), whereeach dominant motion region corresponds to the respective motion trackof a corresponding one of the one or more foreground objects.

In some implementations, the computing system generates a respectiveevent mask for the foreground object corresponding to a first dominantmotion region of the one or more dominant regions based on the firstdominant motion region. The event mask for each object in motion isstored and optionally used to filter the motion event including theobject in motion at a later time.

It should be understood that the particular order in which theoperations in FIGS. 17A-17D have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods and/or processesdescribed herein are also applicable in an analogous manner to themethod 1700 described above with respect to FIGS. 17A-17D.

FIGS. 18A-18D illustrate a flowchart diagram of a method 1800 ofperforming activity recognition for video monitoring on a video serversystem (e.g., the video server system 508, FIG. 5-6) in accordance withsome implementations. FIGS. 18A-18D correspond to instructions stored ina computer memory or computer readable storage medium (e.g., the memory606).

In this method 1800, mathematical processing of motion vectors (e.g.,linear motion vectors) is performed, including clustering and rejectionof false positives. Although the method 1800 occurs on the server, thegeneration of the motion vector may occur locally at the camera or atthe server. The motion vectors are generated in real-time based on livemotion events detected in a live video stream captured by a camera.

In some implementations, a clustering algorithm (e.g., DBscan) is usedin the process. This clustering algorithm allows the growth of clustersinto any shapes. A cluster is promoted as a dense cluster based on itscluster weight, which is in turn based at least partially on the numberof motion vectors contained in it. Only dense clusters are recognized ascategories of recognized events. A user or the server can give acategory name to each category of recognized events. A cluster isupdated when a new vector falls within the range of the cluster. If acluster has not been updated for a long time, the cluster and itsassociated event category is optionally deleted (e.g., via a decayfactor applied to the cluster weight). In some implementations, if acluster remains sparse for a long time, the cluster is optionallydeleted as noise.

As shown in FIG. 18A, at a server (e.g., video server system 508 or theevent categorizer module of the video server system 508) having one ormore processors and memory, the server obtains (1802) a respectivemotion vector for each of a series of motion event candidates inreal-time as said each motion event candidate is detected in a livevideo stream. The motion vector may be received from the cameradirectly, or from an event preparer module of the server. In someimplementations, the server processes a video segment associated with adetected motion event candidate and generates the motion vector.

In response to receiving the respective motion vector for each of theseries of motion event candidates, the server determines (1804) aspatial relationship between the respective motion vector of said eachmotion event candidate to one or more existing clusters establishedbased on a plurality of previously processed motion vectors. This isillustrated in FIGS. 11D-(a)-11D-(e). The existing cluster(s) do notneed to be a dense cluster or have corresponding recognized eventcategory associated with it at this point. When a cluster is not a densecluster, the motion event candidate is associated with a category ofunrecognized events.

In accordance with a determination that the respective motion vector ofa first motion event candidate of the series of motion event candidatesfalls within a respective range of at least a first existing cluster ofthe one or more existing clusters, the server assigns (1806) the firstmotion event candidate to at least a first event category associatedwith the first existing cluster.

In some implementations, the first event category is (1808) a categoryfor unrecognized events. This occurs when the first event category hasnot yet been promoted as a dense cluster and given its own category.

In some implementations, the first event category is (1810) a categoryfor recognized events. This occurs when the first event category hasalready been promoted as a dense cluster and given its own category.

In some implementations, in accordance with a determination that therespective motion vector of a second motion event candidate of theseries of motion event candidates falls beyond a respective range of anyexisting cluster, the server performs (1812) the following operations:assigning the second motion event candidate to a category forunrecognized events; establishing a new cluster for the second motionevent candidate; and associating the new cluster with the category forunrecognized events. This describes a scenario where a new motion vectordoes not fall within any existing cluster in the event space, and thenew motion vector forms its own cluster in the event space. Thecorresponding motion event of the new motion vector is assigned to thecategory for unrecognized events.

In some implementations, the server stores (1814) a respective clustercreation time, a respective current cluster weight, a respective currentcluster center, and a respective current cluster radius for each of theone or more existing clusters. In accordance with the determination thatthe respective motion vector of the first motion event candidate of theseries of motion event candidates falls within the respective range ofthe first existing cluster, the server updates (1816) the respectivecurrent cluster weight, the respective current cluster center, and therespective current cluster radius for the first existing cluster basedon a spatial location of the respective motion vector of the firstmotion event candidate.

In some implementations, before the updating, the first existing clusteris associated with a category of unrecognized events, and after theupdating, the server determines (1818) a respective current clusterdensity for the first existing cluster based on the respective currentcluster weight and the respective current cluster radius of the firstexisting cluster. In accordance with a determination that the respectivecurrent cluster density of the first existing cluster meets apredetermined cluster promotion density threshold, the server promotes(1820) the first existing cluster as a dense cluster. In someimplementations, promoting the first existing cluster further includes(1822) the following operations: creating a new event category for thefirst existing cluster; and disassociating the first existing clusterfrom the category of unrecognized events.

In some implementations, after disassociating the first existing clusterfrom the category of unrecognized events, the server reassigns (1824)all motion vectors in the first existing cluster into the new eventcategory created for the first existing cluster. This describes theretroactive updating of event categories for past motion events, whennew categories are created.

In some implementations, before the updating, the first existing clusteris (1826) associated with a category of unrecognized events, and inaccordance with a determination that the first existing cluster hasincluded fewer than a threshold number of motion vectors for at least athreshold amount of time since the respective cluster creation time ofthe first existing cluster, the server performs (1828) the followingoperations: deleting the first existing cluster including all motionvectors currently in the first existing cluster; and removing the motionevent candidates corresponding to the deleted motion vectors from thecategory of unrecognized events. This describes the pruning of sparseclusters, and motion event candidates in the sparse clusters, forexample, as shown in FIG. 11D-(f). In some implementations, the motionevents are not deleted from the timeline, and are assigned to a categoryof rare events.

In some implementations, the first existing cluster is (1830) associatedwith a category of recognized events, and in accordance with adetermination that the first existing cluster has not been updated forat least a threshold amount of time, the server deletes (1832) the firstexisting cluster including all motion vectors currently in the firstexisting cluster. In some implementations, the server further removes(1834) the motion event candidates corresponding to the deleted motionvectors from the category of recognized events, and deletes (1836) thecategory of recognized events. This describes the retiring of oldinactive clusters. For example, if the camera has been moved to a newlocation, over time, old event categories associated with the previouslocation are automatically eliminated without manual intervention.

In some implementations, the respective motion vector for each of theseries of motion event candidates includes (1838) a start location andan end location of a respective object in motion detected a respectivevideo segment associated with the motion event candidate. The motionvector of this form is extremely compact, reducing processing andtransmission overhead.

In some implementations, to obtain the respective motion vector for eachof the series of motion event candidates in real-time as said eachmotion event candidate is detected in a live video stream, the serverreceives (1840) the respective motion vector for each of the series ofmotion event candidates in real-time from a camera capturing the livevideo stream as said each motion event candidate is detected in the livevideo stream by the camera. In some implementations, the representativemotion vector is a small piece of data received from the camera, wherethe camera has processed the captured video data in real-time andidentified motion event candidate. The camera sends the motion vectorand the corresponding video segment to the server for more sophisticatedprocessing, e.g., event categorization, creating the event mask, etc.

In some implementations, to obtain the respective motion vector for eachof the series of motion event candidates in real-time as said eachmotion event candidate is detected in a live video stream, the serverperforms (1842) the following operations: identifying at least oneobject in motion in a respective video segment associated with themotion event candidate; determining a respective motion track of the atleast one object in motion within a predetermined duration; andgenerating the respective motion vector for the motion event candidatebased on the determined respective motion track of the at least oneobject in motion.

It should be understood that the particular order in which theoperations in FIGS. 18A-18D have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods and/or processesdescribed herein are also applicable in an analogous manner to themethod 1800 described above with respect to FIGS. 18A-18D.

FIGS. 19A-19C illustrate a flowchart diagram of a method 1900 offacilitating review of a video recording (e.g., performing aretrospective event search based on a newly created zone of interest) ona video server system (e.g., video server system 508, FIGS. 5-6) inaccordance with some implementations. FIGS. 19A-19C correspond toinstructions stored in a computer memory or computer readable storagemedium (e.g., the memory 606).

In some implementations, the non-causal (or retrospective) zone searchbased on newly created zones of interest is based on event masks of thepast motion events that have been stored at the server. The eventfiltering based on selected zones of interest can be applied to pastmotion events, and to motion events that are currently being detected inthe live video stream.

As shown in FIG. 19A, the method of facilitating review of a videorecording (e.g., performing a retrospective event search based on anewly created zone of interest) is performed by a server (e.g., thevideo server system 508). The server identifies (1902) a plurality ofmotion events from a video recording, wherein each of the motion eventscorresponds to a respective video segment along a timeline of the videorecording and identifies at least one object in motion within a scenedepicted in the video recording.

The server stores (1904) a respective event mask for each of theplurality of motion events identified in the video recording, therespective event mask including an aggregate of motion pixels associatedwith the at least one object in motion over multiple frames of themotion event. For example, in some implementations, each event includesone object in motion, and corresponds to one event mask. Each scene mayhave multiple motion events occurring at the same time, and havemultiple objects in motion in it.

The server receives (1906) a definition of a zone of interest within thescene depicted in the video recording. In some implementations, thedefinition of the zone of interest is provided by a user or is a defaultzone defined by the server. Receiving the definition of the zone canalso happen when a reviewer is reviewing past events, and has selected aparticular zone that is already defined as an event filter.

In response to receiving the definition of the zone of interest, theserver performs (1908) the following operations: determining, for eachof the plurality of motion events, whether the respective event mask ofthe motion event overlaps with the zone of interest by at least apredetermined overlap factor (e.g., a threshold number of overlappingpixels between the respective event mask and the zone of interest); andidentifying one or more events of interest from the plurality of motionevents, where the respective event mask of each of the identified eventsof interest is determined to overlap with the zone of interest by atleast the predetermined overlap factor. In some implementations, motionevents that touched or entered the zone of interest are identified asevents of interest. The events of interest may be given a colored labelor other visual characteristics associated with the zone of interest,and presented to the reviewer as a group. It is worth noting that thezone of interest is created after the events have already occurred andbeen identified. The fact that the event masks are stored at the timethat the motion events were detected and categorized provides an easyway to go back in time and identify motion events that intersect withthe newly created zone of interest.

In some implementations, the server generates (1910) the respectiveevent mask for each of the plurality of motion events, where thegenerating includes: creating a respective binary motion pixel map foreach frame of the respective video segment associated with the motionevent; and combining the respective binary motion pixel maps of allframes of the respective video segment to generate the respective eventmask for the motion event. As a result, the event mask is a binary mapthat is active (e.g., 1) at all pixel locations where the object inmotion has reached in at least one frame of the video segment. In someimplementations, some other variations of event mask are optionallyused, e.g., giving higher weight to pixel locations that the object inmotion has reached in multiple frames, such that this information may betaken into account when determining the degree of overlap between theevent mask and the zone of interest. More details of the generation ofthe event mask are provided in FIGS. 11C and 11E and accompanyingdescriptions.

In some implementations, the server receives (1912) a first selectioninput from the user to select the zone of interest as a first eventfilter, and visually labels (1914) the identified events of interestwith a respective indicator associated with the zone of interest in anevent review interface. This is illustrated in FIGS. 9L-9N, where Zone A924C is selected by the user, and a past event 922V is identified as anevent of interest for Zone A, and the event indicator of the past event922V is visually labeled by an indicator (e.g., a cross mark) associatedwith Zone A.

In some implementations, the server receives (1916) a second selectioninput selecting one or more object features as a second event filter tobe combined with the first event filter. The server identifies (1918) atleast one motion event from the one or more identified events ofinterest, where the identified at least one motion event includes atleast one object in motion satisfying the one or more object features.The server visually labels (1920) the identified at least one motionevent with a respective indicator associated with both the zone ofinterest and the one or more object features in the event reviewinterface. In some implementations, the one or more object featuresinclude features representing a human being, for example, aspect ratioof the object in motion, movement speed of the object in motion, size ofthe object in motion, shape of the object in motion, etc. The user mayselect to see all events in which a human being entered a particularzone by selecting the zone and the features associated with a humanbeing in an event reviewing interface. The user may also createcombinations of different filters (e.g., zones and/or object features)to create new event filter types.

In some implementations, the definition of the zone of interest includes(1922) a plurality of vertices specified in the scene of the videorecording. In some embodiments, the user is allowed to create zones ofany shapes and sizes by dragging the vertices (e.g., with the dragginggesture in FIGS. 9L-9M). The user may also add or delete one or morevertices from the set of vertices currently shown in the zone definitioninterface.

In some implementations, the server processes (1924) a live video streamdepicting the scene of the video recording to detect a start of a livemotion event, generates (1926) a live event mask based on respectivemotion pixels associated with a respective object in motion identifiedin the live motion event; and determines (1928), in real-time, whetherthe live event mask overlaps with the zone of interest by at least thepredetermined overlap factor. In accordance with a determination thatthe live event mask overlaps with the zone of interest by at least thepredetermined overlap factor, the server generates (1930) a real-timeevent alert for the zone of interest.

In some implementations, the live event mask is generated based on allpast frames in the live motion event that has just been detected. Thelive event mask is updated as each new frame is received. As soon as anoverlap factor determined based on an overlap between the live eventmask and the zone of interest exceeds a predetermined threshold, areal-time alert for the event of interest can be generated and sent tothe user. In a review interface, the visual indicator, for example, acolor, associated with the zone of interest can be applied to the eventindicator for the live motion event. For example, a colored boarder maybe applied to the event indicator on the timeline, and/or the pop-upnotification containing a sprite of the motion event. In someembodiments, the server visually labels (1932) the live motion eventwith a respective indicator associated with the zone of interest in anevent review interface.

It should be understood that the particular order in which theoperations in FIGS. 19A-19C have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods and/or processesdescribed herein are also applicable in an analogous manner to themethod 1900 described above with respect to FIGS. 19A-19C.

FIGS. 20A-20B illustrate a flowchart diagram of a method 2000 ofproviding context-aware zone monitoring on a video server system (e.g.,video server system 508, FIGS. 5-6) in accordance with someimplementations. FIGS. 20A-20B correspond to instructions stored in acomputer memory or computer readable storage medium (e.g., the memory606).

Conventionally, when monitoring a zone of interest within a field ofview of a video surveillance system, the system determines whether anobject has entered the zone of interest based on the image informationwithin the zone of interest. This is ineffective sometimes when theentire zone of interest is obscured by a moving object, and the detailsof the motion (e.g., the trajectory and speed of a moving object) arenot apparent from merely the image within the zone of interest. Forexample, such prior art systems are not be able to distinguish a globallighting change from a object moving in front of the camera andconsequently obscuring the entire view field of the camera. Thetechnique described herein detects motion events without beingconstrained by the zones (i.e., boundaries) that have been defined, andthen determines if a detected event is of interest based on an overlapfactor between the zones and the detected motion events. This allows formore meaningful zone monitoring with context information collectedoutside of the zones of interest.

As shown in FIG. 20A, the method 2000 of monitoring selected zones in ascene depicted in a video stream is performed by a server (e.g., thevideo server system 508). The server receives (2002) a definition of azone of interest within the scene depicted in the video steam. Inresponse to receiving the definition of the zone of interest, the serverdetermines (2004), for each motion event detected in the video stream,whether a respective event mask of the motion event overlaps with thezone of interest by at least a predetermined overlap factor (e.g., athreshold number of pixels), and identifies (2006) the motion event asan event of interest associated with the zone of interest in accordancewith a determination that the respective event mask of the motion eventoverlaps with the zone of interest by at least the predetermined overlapfactor. In other words, the identification of motion events is based onimage information of the whole scene, and then it is determined whetherthe detected motion event is an event of interest based on an overlapfactor between the zone of interest and the event mask of the motionevent.

In some embodiments, the server generates (2008) the respective eventmask for the motion event, where the generating includes: creating arespective binary motion pixel map for each frame of a respective videosegment associated with the motion event; and combining the respectivebinary motion pixel maps of all frames of the respective video segmentto generate the respective event mask for the motion event. Othermethods of generating the event mask are described with respect to FIGS.11C and 11E and accompanying descriptions.

In some embodiments, the server receives (2010) a first selection inputfrom a user to select the zone of interest as a first event filter. Theserver receives (2012) a second selection input from the user to selectone or more object features as a second event filter to be combined withthe first event filter. The server determines (2014) whether theidentified event of interest includes at least one object in motionsatisfying the one or more object features. The server or a componentthereof (e.g., the real-time motion event presentation module 632, FIG.6) generates (2016) a real-time alert for the user in accordance with adetermination that the identified event of interest includes at leastone object in motion satisfying the one or more object features. Forexample, a real-time alert can be generated when an object of interestenters the zone of interest, where the object of interest can be aperson matching the specified object features associated with a humanbeing. In some embodiments, a sub-module (e.g., the personidentification module 626) of the server provides the object featuresassociated with a human being and determines whether the object thatentered the zone of interest is a human being.

In some implementations, the server visually labels (2018) theidentified event of interest with an indicator associated with both thezone of interest and the one or more object features in an event reviewinterface. In some embodiments, the one or more object features are(2020) features representing a human. In some embodiments, thedefinition of the zone of interest includes (2022) a plurality ofvertices specified in the scene of the video recording.

In some embodiments, the video stream is (2024) a live video stream, anddetermining whether the respective event mask of the motion eventoverlaps with the zone of interest by at least a predetermined overlapfactor further includes: processing the live video stream in real-timeto detect a start of a live motion event; generating a live event maskbased on respective motion pixels associated with a respective object inmotion identified in the live motion event; and determining, inreal-time, whether the live event mask overlaps with the zone ofinterest by at least the predetermined overlap factor.

In some embodiments, the server provides (2026) a composite videosegment corresponding to the identified event of interest, the compositevideo segment including a plurality of composite frames each including ahigh-resolution portion covering the zone of interest, and alow-resolution portion covering regions outside of the zone of interest.For example, the high resolution portion can be cropped from theoriginal video stored in the cloud, and the low resolution region can bea stylized abstraction or down-sampled from the original video.

It should be understood that the particular order in which theoperations in FIGS. 20A-20B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods and/or processesdescribed herein are also applicable in an analogous manner to themethod 2000 described above with respect to FIGS. 20A-20B.

For situations in which the systems discussed above collect informationabout users, the users may be provided with an opportunity to opt in/outof programs or features that may collect personal information (e.g.,information about a user's preferences or usage of a smart device). Inaddition, in some implementations, certain data may be anonymized in oneor more ways before it is stored or used, so that personallyidentifiable information is removed. For example, a user's identity maybe anonymized so that the personally identifiable information cannot bedetermined for or associated with the user, and so that user preferencesor user interactions are generalized (for example, generalized based onuser demographics) rather than associated with a particular user.

Although some of various drawings illustrate a number of logical stagesin a particular order, stages that are not order dependent may bereordered and other stages may be combined or broken out. While somereordering or other groupings are specifically mentioned, others will beobvious to those of ordinary skill in the art, so the ordering andgroupings presented herein are not an exhaustive list of alternatives.Moreover, it should be recognized that the stages could be implementedin hardware, firmware, software or any combination thereof.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific implementations. However, theillustrative discussions above are not intended to be exhaustive or tolimit the scope of the claims to the precise forms disclosed. Manymodifications and variations are possible in view of the aboveteachings. The implementations were chosen in order to best explain theprinciples underlying the claims and their practical applications, tothereby enable others skilled in the art to best use the implementationswith various modifications as are suited to the particular usescontemplated.

What is claimed is:
 1. A method, comprising: at a client device having adisplay, memory, and one or more processors, the client device locatedremotely from a video camera of a smart home environment: displaying avideo monitoring user interface on the display, the video monitoringuser interface including a first region for displaying live and/orrecorded video from a video camera of a smart home environment, and asecond region for displaying an event timeline; wherein the eventtimeline includes a plurality of event indicators corresponding torespective events detected within the smart home environment; andwherein the video monitoring user interface includes a live videoaffordance for displaying live video from the video camera; in responseto a user selection of a first event indicator of the plurality of eventindicators: requesting a video feed of the corresponding detected event;and displaying the requested video feed in the first region of the videomonitoring user interface; and in response to a user selection of thelive video affordance: requesting the live video; and displaying thelive video in the first region of the video monitoring user interface.2. The method of claim 1, wherein each event indicator of the pluralityof event indicators has a display property that indicates a category forthe corresponding event.
 3. The method of claim 2, wherein the displayproperty comprises a color of the event indicator.
 4. The method ofclaim 1, wherein each of the plurality of event indicators indicatestiming information of the corresponding event.
 5. The method of claim 4,wherein the timing information includes a start time for thecorresponding event.
 6. The method of claim 1, further comprisingdisplaying a play affordance to control playback of the requested videofeed in the first region.
 7. The method of claim 1, further comprisingdisplaying timing information for the event timeline, includingdisplaying date information for a displayed portion of the eventtimeline.
 8. The method of claim 1, further comprising displaying a nameof the video camera within the video monitoring user interface.
 9. Acomputing system, comprising: a display one or more processors; andmemory storing one or more programs to be executed by the one or moreprocessors, the one or more programs comprising instructions for:displaying a video monitoring user interface on the display, the videomonitoring user interface including a first region for displaying liveand/or recorded video from a video camera of a smart home environment,and a second region for displaying an event timeline; wherein the eventtimeline includes a plurality of event indicators corresponding torespective events detected within the smart home environment; andwherein the video monitoring user interface includes a live videoaffordance for displaying live video from the video camera; in responseto a user selection of a first event indicator of the plurality of eventindicators: requesting a video feed of the corresponding detected event;and displaying the requested video feed in the first region of the videomonitoring user interface; and in response to a user selection of thelive video affordance: requesting the live video; and displaying thelive video in the first region of the video monitoring user interface.10. The computing system of claim 9, further comprising a touch screen;and wherein the user selection of the first event indicator of theplurality of event indicators comprises a user touch at a location onthe touch screen corresponding to the first event indicator.
 11. Thecomputing system of claim 9, wherein displaying the event timelineincludes displaying a plurality of dates and, for each date, displayingevents detected on that date.
 12. The computing system of claim 9,wherein displaying the video monitoring user interface includesdisplaying a live video indicator to indicate to a user whether videoinformation displayed in the first region is live video.
 13. Thecomputing system of claim 9, wherein each event indicator of theplurality of event indicators has a display property that indicates acategory for the corresponding event.
 14. The computing system of claim9, wherein each of the plurality of event indicators indicates timinginformation of the corresponding event.
 15. A non-transitorycomputer-readable storage medium storing one or more programs, the oneor more programs comprising instructions, which, when executed by acomputing system with a display and one or more processors, cause thecomputing system to perform operations comprising: displaying a videomonitoring user interface on the display, the video monitoring userinterface including a first region for displaying live and/or recordedvideo from a video camera of a smart home environment, and a secondregion for displaying an event timeline; wherein the event timelineincludes a plurality of event indicators corresponding to respectiveevents detected within the smart home environment; and wherein the videomonitoring user interface includes a live video affordance fordisplaying live video from the video camera; in response to a userselection of a first event indicator of the plurality of eventindicators: requesting a video feed of the corresponding detected event;and displaying the requested video feed in the first region of the videomonitoring user interface; and in response to a user selection of thelive video affordance: requesting the live video; and displaying thelive video in the first region of the video monitoring user interface.16. The non-transitory computer-readable storage medium of claim 15,wherein each event indicator of the plurality of event indicators has adisplay property that indicates a category for the corresponding event.17. The non-transitory computer-readable storage medium of claim 15,wherein each of the plurality of event indicators indicates timinginformation of the corresponding event.
 18. The non-transitorycomputer-readable storage medium of claim 15, wherein the operationsfurther comprise displaying a pause affordance to pause playback of therequested video feed in the first region.
 19. The non-transitorycomputer-readable storage medium of claim 15, wherein the operationsfurther comprise displaying date information for the plurality of eventindicators.
 20. The non-transitory computer-readable storage medium ofclaim 15, wherein the operations further comprise displaying anaffordance within the video monitoring user interface to generate avideo clip of video displayed in the first region.